00000100:00100000:2.0:1775765647.143842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076349312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765647.143852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765647.143854:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765647.143857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765647.143862:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765647.143882:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765647.143892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765647/real 1775765647] req@ffffa04350930700 x1862023076349440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765647.143905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076349440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765647.143914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765647.143917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765647.143919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765647.143923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765647.143937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765647.143944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765647/real 1775765647] req@ffffa04350931880 x1862023076349568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765647.143952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076349568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765647.143975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765647.143988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765647.143995:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765647.143999:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765647.144019:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765647.144051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765647/real 1775765647] req@ffffa04350932300 x1862023076349696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765647.144064:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076349696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765647.144070:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765647.144072:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765647.144074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076349696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765648.167112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765649.191097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765649.191166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765650.215131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765651.239156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765652.263291:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765652.263587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076349824:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765652.263709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.263778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350932300 x1862023076349824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.263873:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076349824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.263944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.263974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.263987:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076349824:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765652.264115:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076349952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765652.264212:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.264255:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350931880 x1862023076349952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.264314:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076349952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.264331:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.264355:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.264386:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076349952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765652.264416:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765652.264476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.264533:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350930700 x1862023076350080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.264592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076350080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.264605:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.264610:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.264614:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765652.264622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765652.264662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.264676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350931c00 x1862023076350208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.264691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076350208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.264701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.264705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.264709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765652.264732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765652.264758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.264771:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350932a00 x1862023076350336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.264784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076350336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.264795:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.264799:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.264802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076350336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765652.264810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765652.264833:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.264846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350930000 x1862023076350464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.264894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076350464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.264926:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.264933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.264955:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765652.264963:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765652.264989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.265018:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350930a80 x1862023076350592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.265034:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076350592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.265045:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.265049:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.265054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765652.265109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765652.265151:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.265167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350932680 x1862023076350720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.265183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076350720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.265237:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.265244:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.265250:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765652.265264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765652.265309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765652.265323:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765652/real 1775765652] req@ffffa04350930e00 x1862023076350848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765652.265337:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076350848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765652.265347:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765652.265351:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765652.265355:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076350848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765653.287158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765654.311181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765655.335370:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765657.383317:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765657.383526:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076350976:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765657.383630:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.383708:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350930e00 x1862023076350976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.383826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076350976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.383992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076350976:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765657.384147:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765657.384213:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.384251:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350932680 x1862023076351104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.384296:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076351104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.384347:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384367:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384384:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765657.384405:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765657.384453:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.384479:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350930a80 x1862023076351232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.384521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076351232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.384533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765657.384569:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765657.384598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.384612:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350930000 x1862023076351360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.384627:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076351360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.384637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384645:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765657.384656:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765657.384691:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.384712:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350932a00 x1862023076351488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.384737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076351488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.384755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076351488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765657.384780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765657.384812:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.384833:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350931c00 x1862023076351616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.384900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076351616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.384918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.384925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.384932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765657.384971:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765657.385004:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.385019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350930700 x1862023076351744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.385071:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076351744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.385082:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.385086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.385090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765657.385099:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765657.385122:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.385134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350931880 x1862023076351872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.385147:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076351872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.385179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.385185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.385191:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076351872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765657.385204:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765657.385248:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765657.385270:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765657/real 1775765657] req@ffffa04350932300 x1862023076352000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765657.385294:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076352000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765657.385308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765657.385314:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765657.385321:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765658.407106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765659.431157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765660.455105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765661.479176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765662.503208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765662.503431:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076352128:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765662.503528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.503590:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350932300 x1862023076352128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.503625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076352128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.503699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.503709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.503723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076352128:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765662.503763:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765662.503845:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.503892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350931880 x1862023076352256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.503928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076352256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.503964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.503977:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.503989:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765662.504007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765662.504085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350930700 x1862023076352384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504131:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076352384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504191:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765662.504229:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765662.504273:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504303:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350931c00 x1862023076352512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504329:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076352512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504373:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504385:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765662.504431:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765662.504456:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504467:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350932a00 x1862023076352640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504481:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076352640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504491:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504495:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504498:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076352640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765662.504505:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765662.504524:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504534:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350930000 x1862023076352768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504560:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076352768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504574:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504578:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765662.504585:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765662.504607:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504618:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350930a80 x1862023076352896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504632:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076352896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504641:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504644:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504648:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076352896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765662.504654:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076353024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765662.504671:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504681:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350932680 x1862023076353024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076353024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504724:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504728:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076353024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765662.504735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076353152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765662.504758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765662.504769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765662/real 1775765662] req@ffffa04350930e00 x1862023076353152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765662.504782:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076353152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765662.504791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765662.504794:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765662.504798:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076353152:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775765667.623125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765667.623138:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623147:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623157:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623165:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765667.623199:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076353280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765667.623243:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623248:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623252:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623257:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623266:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765667.623277:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076353408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623287:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623290:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765667.623292:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623295:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623300:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623303:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623307:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765667.623315:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076353536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623323:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623326:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765667.623327:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623331:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623334:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623337:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623347:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765667.623356:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076353664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765667.623371:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623374:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623378:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623381:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623386:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765667.623394:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076353792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765667.623406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623410:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623413:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623417:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623421:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765667.623430:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076353920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623443:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765667.623450:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623454:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623457:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623461:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623466:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765667.623476:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076354048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765667.623491:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623494:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623497:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623501:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765667.623515:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076354176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765667.623525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765667.623529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765667.623531:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765667.623534:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2169 00000100:00080000:2.0:1775765667.623538:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765667.623542:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765667.623546:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765667.623561:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076354304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765669.671300:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765671.719318:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765672.743183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765672.743204:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743220:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743242:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743254:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765672.743314:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076354432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743354:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765672.743356:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743362:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743368:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743379:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.743416:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765672.743440:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076354560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743461:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765672.743464:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743495:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743503:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743509:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.743517:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765672.743541:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076354688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743581:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765672.743585:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743596:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743609:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743622:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.743642:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765672.743673:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076354816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743696:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743706:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765672.743740:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743748:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743754:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743774:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.743784:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765672.743809:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076354944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743841:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765672.743861:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743872:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743884:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.743889:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.743896:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765672.743917:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076355072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.743955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.743965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765672.743969:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.743986:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.743993:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.744003:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.744018:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765672.744095:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076355200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.744111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.744120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765672.744124:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.744131:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.744155:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.744169:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.744181:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765672.744201:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076355328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765672.744246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765672.744252:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765672.744258:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765672.744269:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2174 00000100:00080000:2.0:1775765672.744281:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765672.744292:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765672.744303:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765672.744320:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076355456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765673.767312:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765675.815202:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765677.863188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765677.863204:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863218:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863232:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863241:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765677.863295:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076355584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863341:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863345:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765677.863348:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863352:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863356:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863361:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863374:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765677.863395:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076355712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863409:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765677.863426:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863433:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863437:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863441:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863447:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765677.863461:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076355840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863471:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863490:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765677.863492:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863496:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863505:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863510:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863516:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765677.863529:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076355968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863538:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863541:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765677.863549:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863564:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863571:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863575:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863578:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765677.863593:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076356096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863608:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765677.863626:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863636:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863641:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863647:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863678:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765677.863707:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076356224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863757:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765677.863760:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863768:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863774:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863779:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863790:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765677.863811:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076356352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765677.863842:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863851:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863865:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.863886:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.863901:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765677.863934:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076356480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765677.863944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765677.863956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765677.863963:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765677.863980:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2179 00000100:00080000:2.0:1775765677.863995:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765677.864004:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765677.864015:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765677.864028:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076356608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765679.079132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765680.103085:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765681.127184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765682.151175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775765682.983128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765682.983140:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983149:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983162:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983189:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765682.983223:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076356736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765682.983249:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983252:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983255:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983258:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983265:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765682.983276:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076356864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983310:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765682.983312:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983317:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983322:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983327:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983333:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765682.983349:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076356992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983367:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765682.983369:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983373:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983377:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983380:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983385:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765682.983399:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076357120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983442:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765682.983453:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983457:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983470:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983474:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983479:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765682.983496:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076357248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983505:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983509:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765682.983511:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983516:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983521:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983528:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983553:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765682.983567:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076357376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983583:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983585:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765682.983587:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983591:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983595:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983600:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983605:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765682.983635:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076357504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765682.983685:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983692:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983699:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983707:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983713:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765682.983724:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076357632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765682.983737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765682.983740:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765682.983741:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765682.983747:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2184 00000100:00080000:2.0:1775765682.983750:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765682.983754:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765682.983761:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765682.983774:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076357760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765683.175093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765684.199114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775765685.031287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765685.223154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765686.247135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765687.271112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765688.295121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765689.319112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765690.343173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765691.367096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765692.391132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775765693.223169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765693.223184:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223194:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223205:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223238:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765693.223273:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076359040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765693.223307:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223311:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223315:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223320:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223329:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765693.223344:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076359168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765693.223380:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223386:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223391:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223395:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223402:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765693.223416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076359296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765693.223433:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223437:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223442:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223448:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765693.223473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076359424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765693.223500:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223504:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223508:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223511:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223531:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765693.223551:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076359552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765693.223577:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223583:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223593:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223599:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223607:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765693.223621:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076359680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223648:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765693.223650:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223656:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223664:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223669:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765693.223694:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076359808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223717:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223721:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765693.223724:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223737:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223742:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223748:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223753:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765693.223772:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076359936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765693.223805:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765693.223811:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765693.223815:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765693.223821:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2194 00000100:00080000:2.0:1775765693.223826:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765693.223832:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765693.223839:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765693.223853:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076360064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765693.415094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765694.439150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765695.463094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765696.487097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765697.511098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775765698.343162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765698.343172:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343181:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343190:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343212:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765698.343253:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076360192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765698.343279:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343282:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343286:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343290:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765698.343309:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076360320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765698.343321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343327:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343329:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343333:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765698.343339:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076360448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765698.343354:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343357:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343361:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343363:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343368:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765698.343379:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076360576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343387:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765698.343390:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343392:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343394:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343397:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765698.343409:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076360704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343417:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765698.343426:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343458:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343464:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343483:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343492:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765698.343523:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076360832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765698.343536:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343539:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343544:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343553:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765698.343564:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076360960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765698.343574:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343577:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343579:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343584:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343591:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765698.343623:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076361088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765698.343631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765698.343633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765698.343635:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765698.343648:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2200 00000100:00080000:2.0:1775765698.343654:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765698.343661:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765698.343667:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765698.343682:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076361216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765698.535176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765699.559110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765700.583100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765701.607086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765702.631099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765703.655149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765704.679142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765705.703131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765706.535256:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765706.727108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765707.559149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765707.751095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765708.583165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765708.775141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765709.607319:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765709.799179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765710.631215:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765710.823142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765711.655254:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765711.847121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765712.679096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765712.871120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765713.703165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765713.895126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765714.919138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765715.943134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765716.967113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765717.991158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775765728.039287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765729.063275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063282:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765729.063287:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063298:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063308:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063313:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765729.063348:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076367104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765729.063376:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063379:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063382:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063386:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063394:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765729.063415:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076367232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063424:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765729.063428:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063446:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063457:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063460:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063466:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765729.063481:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076367360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063492:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765729.063493:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063507:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063510:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063514:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765729.063530:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076367488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765729.063547:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063556:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063558:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063560:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063563:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765729.063575:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076367616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765729.063605:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063609:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063612:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063615:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063622:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765729.063635:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076367744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765729.063664:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063668:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063672:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063679:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063687:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765729.063698:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076367872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063719:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765729.063724:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063733:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063743:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063748:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063753:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765729.063765:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076368000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765729.063796:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765729.063801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765729.063805:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765729.063814:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2230 00000100:00080000:2.0:1775765729.063817:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765729.063825:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765729.063832:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765729.063841:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076368128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765730.087228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765732.135213:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765734.183137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765734.183154:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183166:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183180:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183187:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765734.183221:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076368256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765734.183264:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183267:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183271:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183278:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183296:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765734.183311:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076368384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765734.183324:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183326:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183335:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183339:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765734.183348:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076368512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183357:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765734.183359:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183362:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183371:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183378:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183383:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765734.183397:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076368640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183414:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765734.183418:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183425:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183428:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183430:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183433:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765734.183444:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076368768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765734.183465:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183472:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183479:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183484:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183490:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765734.183510:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076368896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765734.183557:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183565:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183571:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183575:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183582:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765734.183601:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076369024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183632:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765734.183635:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183643:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183648:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183650:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183659:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765734.183673:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076369152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765734.183690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765734.183693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765734.183694:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765734.183698:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2235 00000100:00080000:2.0:1775765734.183707:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765734.183712:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765734.183718:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765734.183731:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076369280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765736.231249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765737.255168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765738.279323:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765739.303187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765739.303203:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303214:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303225:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303234:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765739.303283:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076369408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765739.303327:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303333:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303339:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303346:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303360:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765739.303395:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076369536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765739.303431:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303435:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303451:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303455:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303461:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765739.303488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076369664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765739.303502:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303506:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303509:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303513:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303519:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765739.303532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076369792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303576:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303579:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765739.303581:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303585:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303599:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303609:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303616:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765739.303632:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076369920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765739.303656:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303660:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303664:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303672:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303680:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765739.303693:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076370048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765739.303715:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303731:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303739:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303747:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303751:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765739.303767:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076370176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303810:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303814:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765739.303817:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303824:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303834:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303840:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303849:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765739.303866:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076370304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765739.303909:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765739.303913:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765739.303916:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765739.303925:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2240 00000100:00080000:2.0:1775765739.303931:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765739.303939:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765739.303944:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765739.303960:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076370432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765748.519182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765749.543357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765749.543376:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543388:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543423:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543440:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765749.543478:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076371712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543510:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765749.543516:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543522:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543530:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543534:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543544:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765749.543559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076371840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765749.543590:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543599:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543604:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543607:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543612:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765749.543625:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076371968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765749.543649:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543666:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543672:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543676:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543682:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765749.543699:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076372096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543708:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765749.543724:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543743:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543747:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543751:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543757:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765749.543775:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076372224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543809:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543827:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765749.543829:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543837:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543840:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543844:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543870:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765749.543885:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076372352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.543910:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.543921:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765749.543928:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.543945:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.543950:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.543962:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.543994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765749.544021:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076372480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.544068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.544074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765749.544077:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.544087:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.544093:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.544097:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.544106:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765749.544122:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076372608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765749.544136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765749.544145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765749.544154:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765749.544169:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2251 00000100:00080000:2.0:1775765749.544183:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765749.544196:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765749.544201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765749.544225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076372736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765750.567180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765752.615147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765754.663223:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765755.687178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765759.783163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765760.807165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765761.831199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765762.855135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765763.879179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765764.903167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765765.927165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765766.951140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765767.975256:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765770.023216:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765771.047198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765775.143258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765775.143274:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143288:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143300:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143309:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765775.143362:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076377472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143410:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765775.143412:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143419:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143427:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143434:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.143448:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765775.143470:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076377600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143507:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765775.143510:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143517:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143525:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143532:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.143540:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765775.143558:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076377728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143585:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143592:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765775.143596:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143613:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143632:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143640:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.143649:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765775.143674:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076377856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143708:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765775.143713:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143727:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143780:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143789:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.143797:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765775.143820:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076377984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765775.143849:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.143860:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.143871:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.143879:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.143889:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765775.143912:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076378112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.143971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.143982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765775.143987:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.144000:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.144012:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.144024:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.144064:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765775.144084:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076378240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.144097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.144102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765775.144104:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.144109:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.144115:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.144120:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.144126:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765775.144150:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076378368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765775.144165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765775.144169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765775.144172:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765775.144177:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2276 00000100:00080000:2.0:1775765775.144192:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765775.144199:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765775.144216:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765775.144231:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076378496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765778.215187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765785.383255:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765786.407148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765787.431138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765788.455107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765789.479219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765790.503186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765791.527126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765792.551105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765793.575131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765793.575189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765794.599093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765795.623098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765795.623145:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765796.647202:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765797.671176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765798.695125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765800.743266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743289:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765800.743298:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743317:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743338:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743350:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765800.743449:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076383232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743529:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743538:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765800.743542:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743551:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743560:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743569:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.743589:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765800.743617:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076383360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743639:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765800.743651:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743659:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743666:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743675:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.743684:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765800.743711:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076383488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765800.743734:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743751:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743761:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743767:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.743774:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765800.743794:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076383616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743821:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765800.743828:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743836:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743841:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743846:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.743852:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765800.743865:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076383744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765800.743884:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743892:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743897:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.743905:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.743911:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765800.743925:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076383872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.743965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.743970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765800.743974:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.743986:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.743997:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.744010:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.744017:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765800.744092:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076384000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.744113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.744120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765800.744123:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.744134:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.744141:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.744147:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.744155:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765800.744180:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076384128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765800.744205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765800.744222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765800.744228:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765800.744238:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2302 00000100:00080000:2.0:1775765800.744262:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765800.744278:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765800.744291:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765800.744315:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076384256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765801.767216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765803.815270:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765805.863188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765815.079238:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765816.103178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765816.103196:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103210:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103224:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103235:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765816.103299:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076386688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765816.103371:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103378:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103384:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103389:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765816.103424:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076386816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765816.103474:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103481:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103489:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103499:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103509:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765816.103530:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076386944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765816.103556:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103565:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103570:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103579:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103591:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765816.103608:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076387072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765816.103646:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103660:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103679:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103686:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103692:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765816.103707:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076387200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765816.103732:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103738:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103743:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103753:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103759:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765816.103776:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076387328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103804:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103810:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765816.103815:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103826:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103834:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103842:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103848:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765816.103862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076387456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103894:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103897:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765816.103899:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103905:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103913:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103917:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.103926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765816.103942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076387584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765816.103968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765816.103975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765816.103977:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765816.103983:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2317 00000100:00080000:2.0:1775765816.103990:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765816.103999:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765816.104004:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765816.104016:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076387712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765817.127225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765819.175175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765821.223395:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076387840:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765821.223464:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223505:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350930e00 x1862023076387840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076387840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223538:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076387840:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765821.223563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076387968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765821.223594:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223607:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350930000 x1862023076387968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223620:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076387968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223654:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076387968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765821.223662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765821.223683:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223695:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350930a80 x1862023076388096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223710:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076388096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223744:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765821.223767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765821.223791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223805:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350932680 x1862023076388224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223836:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076388224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223845:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765821.223857:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765821.223873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350931880 x1862023076388352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076388352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076388352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765821.223920:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765821.223937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.223944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350930700 x1862023076388480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.223952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076388480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.223960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.223964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.223968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765821.223973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765821.223992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.224002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350931c00 x1862023076388608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.224015:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076388608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.224022:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.224025:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.224028:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765821.224034:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765821.224056:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.224098:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350932a00 x1862023076388736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.224112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076388736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.224118:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.224120:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.224122:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765821.224126:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765821.224160:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765821.224169:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765821/real 1775765821] req@ffffa04350932300 x1862023076388864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765821.224179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076388864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765821.224186:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765821.224188:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765821.224190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076388864:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775765826.343154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765826.343165:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343175:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343184:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343189:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765826.343235:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076388992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765826.343283:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343287:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343290:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343294:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343301:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765826.343312:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076389120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765826.343325:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343330:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343333:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343335:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765826.343345:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076389248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765826.343356:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343365:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343368:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343371:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765826.343384:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076389376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765826.343394:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343397:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343400:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343401:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343404:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765826.343412:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076389504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765826.343424:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343427:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343430:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343432:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343435:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765826.343442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076389632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343466:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765826.343469:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343476:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343480:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343483:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343488:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765826.343509:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076389760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765826.343520:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343522:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343524:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343527:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343532:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765826.343543:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076389888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765826.343551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765826.343554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765826.343557:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765826.343561:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2328 00000100:00080000:2.0:1775765826.343570:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765826.343575:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765826.343583:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765826.343594:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076390016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765830.439302:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765831.463192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765831.463238:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463258:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463280:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463293:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765831.463343:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076390144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463386:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765831.463389:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463394:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463400:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463405:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463418:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765831.463435:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076390272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765831.463499:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463508:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463533:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463539:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463547:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765831.463580:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076390400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463599:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765831.463602:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463608:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463614:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463623:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463635:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765831.463658:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076390528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463703:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765831.463706:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463723:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463729:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463742:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463760:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765831.463781:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076390656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463801:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463806:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765831.463809:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463819:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463827:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463832:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463845:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765831.463862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076390784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.463908:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.463917:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765831.463919:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.463930:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.463940:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.463950:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.463960:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765831.463979:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076390912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.464002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.464007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765831.464019:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.464026:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.464034:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.464085:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.464099:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765831.464127:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076391040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765831.464162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765831.464175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765831.464178:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765831.464186:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2333 00000100:00080000:2.0:1775765831.464219:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765831.464229:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765831.464240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765831.464260:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076391168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765832.487243:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765834.535272:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765836.583247:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765836.583260:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583271:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583284:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583293:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765836.583364:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076391296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765836.583423:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583428:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583435:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583441:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583452:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765836.583469:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076391424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765836.583497:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583502:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583505:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583508:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583514:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765836.583539:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076391552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765836.583555:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583568:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583574:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583578:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583585:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765836.583614:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076391680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765836.583629:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583634:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583639:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583645:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583654:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765836.583676:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076391808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583720:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765836.583723:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583731:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583738:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583750:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583762:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765836.583795:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076391936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765836.583851:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583878:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583887:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583899:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583910:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765836.583932:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076392064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.583951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.583956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765836.583958:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.583962:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.583968:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.583971:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.583978:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765836.583991:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076392192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765836.584002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765836.584008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765836.584012:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765836.584021:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2338 00000100:00080000:2.0:1775765836.584041:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765836.584046:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765836.584054:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765836.584115:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076392320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765839.655471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7fb80 x1862023076392320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655511:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076392320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655550:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076392320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765839.655559:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7e680 x1862023076392064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655573:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076392064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655585:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655588:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076392064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765839.655596:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7f800 x1862023076391808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076391808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076391808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765839.655659:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7c700 x1862023076391936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655670:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076391936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076391936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765839.655690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7ed80 x1862023076391424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076391424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655708:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655712:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076391424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765839.655721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7c000 x1862023076391552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076391552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655750:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655758:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076391552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765839.655764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7e300 x1862023076391680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076391680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655784:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655788:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076391680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765839.655799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7dc00 x1862023076391296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076391296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076391296:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765839.655864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765836/real 1775765839] req@ffffa0434aa7ce00 x1862023076392192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765839.655886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076392192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765839.655895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765839.655899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765839.655902:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076392192:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775765841.703176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765841.703189:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703199:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703208:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703215:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765841.703248:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076392448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703279:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765841.703281:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703286:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703290:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703294:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703305:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765841.703320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076392576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765841.703374:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703379:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703384:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703388:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703394:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765841.703416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076392704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765841.703433:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703438:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703445:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703453:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703483:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765841.703499:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076392832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765841.703527:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703536:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703555:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703560:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703567:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765841.703584:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076392960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765841.703616:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703623:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703627:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703632:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703638:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765841.703655:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076393088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703668:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765841.703674:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703681:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703689:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703696:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703703:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765841.703718:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076393216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765841.703795:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703802:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703810:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703815:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703821:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765841.703837:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076393344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765841.703850:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765841.703854:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765841.703857:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765841.703862:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2343 00000100:00080000:2.0:1775765841.703869:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765841.703877:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765841.703888:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765841.703903:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076393472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765842.727176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765843.751190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765844.775178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765845.799180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765846.823200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765847.847215:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765848.871242:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765851.943480:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076394752:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765851.943624:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.943710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825d880 x1862023076394752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.943771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076394752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.943820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.943840:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.943853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076394752:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765851.943872:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076394880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765851.943929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.943963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825ce00 x1862023076394880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944125:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076394880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944218:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944237:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944258:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076394880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765851.944325:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765851.944389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825ea00 x1862023076395008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944467:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076395008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944485:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944493:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944499:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765851.944516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765851.944547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944565:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825e680 x1862023076395136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944582:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076395136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944597:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765851.944609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765851.944628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825ed80 x1862023076395264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076395264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944682:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076395264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765851.944694:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765851.944737:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944760:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825dc00 x1862023076395392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076395392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765851.944826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765851.944858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825c000 x1862023076395520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.944888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076395520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.944898:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.944902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.944917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765851.944945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765851.944973:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.944986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825f100 x1862023076395648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.945058:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076395648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.945071:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.945076:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.945081:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765851.945090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765851.945121:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765851.945136:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765851/real 1775765851] req@ffffa0435825d500 x1862023076395776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765851.945151:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076395776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765851.945161:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765851.945166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765851.945169:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076395776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765855.015104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765856.039123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765857.063113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765857.063166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765857.063175:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063183:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063192:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063198:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765857.063231:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076395904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765857.063262:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063267:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063271:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063276:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063287:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765857.063304:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076396032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765857.063330:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063336:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063342:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063357:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063374:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765857.063394:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076396160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063407:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063410:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765857.063412:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063417:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063423:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063430:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063438:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765857.063455:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076396288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063491:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063496:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765857.063498:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063504:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063510:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063517:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063522:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765857.063535:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076396416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063557:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765857.063572:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063580:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063588:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063612:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063623:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765857.063641:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076396544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063655:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063658:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765857.063661:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063670:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063676:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063683:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063690:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765857.063710:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076396672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765857.063729:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063739:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063743:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063751:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063760:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765857.063783:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076396800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765857.063796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765857.063800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765857.063802:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765857.063809:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2358 00000100:00080000:2.0:1775765857.063816:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765857.063821:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765857.063827:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765857.063843:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076396928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765858.087124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765859.111100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765860.135194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765861.159117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765862.183139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765862.183151:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183161:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183172:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183182:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765862.183222:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076397056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765862.183267:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183271:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183275:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183279:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183299:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765862.183311:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076397184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183319:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183322:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765862.183323:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183326:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183328:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183330:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183334:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765862.183342:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076397312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183351:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765862.183353:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183360:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183363:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183366:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183369:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765862.183383:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076397440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183400:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183402:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765862.183403:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183406:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183409:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183411:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183414:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765862.183423:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076397568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183445:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765862.183449:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183452:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183454:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183468:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183474:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765862.183485:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076397696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183500:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183502:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765862.183504:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183513:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183517:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183522:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183526:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765862.183538:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076397824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183546:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765862.183552:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183558:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183562:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183564:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183568:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765862.183578:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076397952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765862.183590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765862.183597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765862.183600:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765862.183606:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2363 00000100:00080000:2.0:1775765862.183612:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765862.183618:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765862.183623:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765862.183630:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076398080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765864.231148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765867.303132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765867.303147:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303156:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303165:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303172:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765867.303207:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076398208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765867.303237:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303242:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303246:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303251:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303262:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765867.303281:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076398336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765867.303297:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303300:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303304:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303324:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303330:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765867.303340:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076398464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303346:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765867.303350:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303353:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303357:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303360:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303364:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765867.303374:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076398592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303386:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303389:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765867.303390:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303395:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303398:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303404:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303410:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765867.303419:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076398720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303431:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303438:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765867.303440:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303444:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303447:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303452:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303460:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765867.303472:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076398848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303481:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303487:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765867.303489:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303496:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303503:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303511:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303519:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765867.303535:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076398976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303555:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765867.303560:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303564:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303568:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303573:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303585:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765867.303599:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076399104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765867.303620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765867.303625:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765867.303628:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765867.303635:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2368 00000100:00080000:2.0:1775765867.303643:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765867.303649:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765867.303668:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765867.303687:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076399232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765868.327228:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765870.375237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765872.423107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765873.447152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765874.471104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765875.495104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765876.519105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765876.519177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765877.543119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765877.543324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076400512:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765877.543390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543437:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350932300 x1862023076400512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543475:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076400512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543501:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543509:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543514:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076400512:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765877.543539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765877.543567:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350932a00 x1862023076400640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076400640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543630:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543638:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765877.543652:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765877.543673:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543683:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350931c00 x1862023076400768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543703:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076400768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543742:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765877.543779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765877.543801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350930700 x1862023076400896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076400896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543828:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543832:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076400896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765877.543846:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765877.543856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350931880 x1862023076401024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543869:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076401024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765877.543888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765877.543901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543908:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350932680 x1862023076401152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076401152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765877.543927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765877.543936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350930a80 x1862023076401280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.543948:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076401280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.543953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.543955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.543957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765877.543962:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765877.543979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.543998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350930000 x1862023076401408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.544012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076401408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.544021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.544024:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.544047:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765877.544052:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765877.544068:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765877.544075:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765877/real 1775765877] req@ffffa04350930e00 x1862023076401536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765877.544083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076401536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765877.544089:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765877.544091:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765877.544092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076401536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765882.663272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076401664:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765882.663335:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663379:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350930e00 x1862023076401664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663406:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076401664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076401664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765882.663471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765882.663501:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350930000 x1862023076401792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663556:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076401792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765882.663608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765882.663634:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350930a80 x1862023076401920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663672:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076401920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663683:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663687:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076401920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765882.663704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765882.663742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350932680 x1862023076402048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076402048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663774:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663777:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663780:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765882.663786:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765882.663801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350931880 x1862023076402176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076402176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663836:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765882.663865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765882.663884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350930700 x1862023076402304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076402304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765882.663933:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765882.663950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.663959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350931c00 x1862023076402432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.663970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076402432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.663979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.663982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.663985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765882.663990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765882.664004:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.664012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350932a00 x1862023076402560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.664049:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076402560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.664073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.664077:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.664080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765882.664086:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765882.664105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765882.664115:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765882/real 1775765882] req@ffffa04350932300 x1862023076402688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765882.664127:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076402688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765882.664135:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765882.664138:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765882.664141:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076402688:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775765885.735341:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765887.783138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765887.783280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076402816:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775765887.783352:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783398:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350932300 x1862023076402816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783424:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076402816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783444:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783461:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783472:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076402816:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775765887.783488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765887.783516:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783527:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350932a00 x1862023076402944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783535:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076402944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783541:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783544:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076402944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765887.783564:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765887.783575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783581:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350931c00 x1862023076403072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783588:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076403072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783594:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783596:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765887.783601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765887.783620:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783626:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350930700 x1862023076403200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076403200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783639:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765887.783647:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765887.783677:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350931880 x1862023076403328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783701:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076403328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783711:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783715:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783718:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076403328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765887.783724:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765887.783741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783748:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350932680 x1862023076403456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076403456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765887.783779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775765887.783791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783797:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350930a80 x1862023076403584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076403584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775765887.783815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765887.783824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350930000 x1862023076403712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783836:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076403712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783851:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775765887.783857:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775765887.783870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775765887.783876:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765887/real 1775765887] req@ffffa04350930e00 x1862023076403840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765887.783883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076403840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765887.783887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765887.783889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775765887.783891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076403840:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775765892.903164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765892.903177:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903186:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903196:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903204:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765892.903237:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076403968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765892.903275:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903289:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903292:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903298:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903307:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765892.903325:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076404096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765892.903341:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903344:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903349:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903351:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903358:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765892.903373:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076404224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903397:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765892.903400:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903409:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903418:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903424:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903436:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765892.903451:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076404352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765892.903465:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903473:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903477:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903479:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903482:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765892.903487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076404480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765892.903498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903500:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903501:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903504:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903506:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765892.903511:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076404608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765892.903523:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903527:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903529:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903532:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903535:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765892.903545:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076404736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765892.903557:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903563:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903566:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903569:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903572:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765892.903579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076404864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765892.903589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765892.903592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765892.903594:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765892.903598:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2394 00000100:00080000:2.0:1775765892.903601:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765892.903604:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765892.903624:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765892.903646:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076404992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765893.927149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765895.975157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765898.023195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765898.023206:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023228:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023239:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023247:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765898.023291:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076405120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765898.023327:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023332:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023344:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023352:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023362:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765898.023380:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076405248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765898.023398:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023427:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023433:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023441:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765898.023461:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076405376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765898.023477:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023482:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023498:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023507:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023516:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765898.023532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076405504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765898.023571:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023580:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023584:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023588:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023592:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765898.023603:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076405632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023618:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023621:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765898.023625:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023632:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023637:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023642:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023649:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765898.023662:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076405760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023680:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765898.023681:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023685:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023691:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023699:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023708:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765898.023724:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076405888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023748:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765898.023751:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023760:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023773:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023782:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023789:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765898.023803:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076406016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765898.023814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765898.023821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765898.023825:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765898.023832:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2399 00000100:00080000:2.0:1775765898.023837:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765898.023844:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765898.023849:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765898.023861:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076406144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765899.047204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765900.071328:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765902.119267:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765903.143137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765903.143151:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143161:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143172:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143180:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765903.143214:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076406272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765903.143254:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143258:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143261:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765903.143283:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076406400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765903.143302:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143310:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143312:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143314:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143317:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765903.143326:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076406528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143335:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765903.143336:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143339:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143346:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143350:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143354:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765903.143363:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076406656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765903.143375:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143378:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143381:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143384:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143389:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765903.143402:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076406784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143409:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765903.143415:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143423:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143427:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143429:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143433:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765903.143442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076406912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765903.143473:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143478:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143482:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143488:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143493:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765903.143505:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076407040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765903.143534:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143543:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143546:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143549:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143553:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765903.143571:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076407168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765903.143588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765903.143591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765903.143594:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765903.143599:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2404 00000100:00080000:2.0:1775765903.143613:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765903.143620:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765903.143626:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765903.143640:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076407296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775765904.167240:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765906.215298:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775765906.215367:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775765906.215403:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765906.215415:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215465:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215508:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215527:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215544:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215562:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215581:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215602:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765906.215636:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:2.0:1775765908.263172:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765908.263184:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263195:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263204:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263212:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765908.263242:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076407424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263283:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765908.263285:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263290:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263295:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263299:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263309:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765908.263322:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076407552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765908.263337:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263340:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263343:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263347:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263351:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765908.263359:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076407680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263372:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765908.263374:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263377:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263388:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263393:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263398:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765908.263407:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076407808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263419:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263422:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765908.263424:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263428:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263432:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263436:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263441:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765908.263450:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076407936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263461:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765908.263479:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263483:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263487:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263491:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263496:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765908.263506:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076408064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263529:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263532:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765908.263534:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263538:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263542:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263547:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263552:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765908.263563:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076408192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263573:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263576:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765908.263578:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263581:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263585:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263589:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263593:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765908.263604:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076408320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765908.263620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765908.263624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765908.263626:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765908.263631:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2409 00000100:00080000:2.0:1775765908.263639:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765908.263643:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765908.263648:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765908.263659:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076408448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765913.383142:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383157:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383171:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383179:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765913.383237:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076408576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765913.383264:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383267:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383271:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383275:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383295:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765913.383308:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076408704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383319:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383337:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765913.383339:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383343:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383356:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383360:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383365:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765913.383380:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076408832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383393:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765913.383395:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383397:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383400:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383403:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383407:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765913.383432:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076408960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383443:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765913.383445:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383451:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383485:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383494:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383507:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765913.383520:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076409088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383538:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383544:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765913.383548:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383555:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383562:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383566:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383571:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765913.383580:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076409216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383606:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383611:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765913.383616:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383621:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383624:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383627:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383632:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765913.383643:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076409344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383659:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765913.383669:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383675:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383681:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383690:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383697:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765913.383727:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076409472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765913.383736:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765913.383746:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765913.383749:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765913.383758:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2415 00000100:00080000:2.0:1775765913.383764:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765913.383769:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765913.383773:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765913.383793:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076409600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765916.455172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765917.479116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765918.503141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765918.503160:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503169:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503180:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503187:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765918.503240:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076409728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503267:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765918.503269:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503272:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503276:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503280:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503289:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765918.503301:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076409856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765918.503320:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503323:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503325:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503327:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503331:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765918.503358:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076409984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503389:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503393:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765918.503395:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503426:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503432:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503440:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503449:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765918.503465:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076410112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503477:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503481:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765918.503483:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503486:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503490:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503493:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503500:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765918.503516:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076410240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503526:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765918.503539:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503545:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503548:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503554:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503560:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765918.503571:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076410368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503595:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765918.503597:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503602:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503606:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503609:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503614:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765918.503631:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076410496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503651:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503655:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765918.503656:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503660:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503665:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503670:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503673:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765918.503681:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076410624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765918.503690:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765918.503697:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765918.503701:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765918.503709:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2420 00000100:00080000:2.0:1775765918.503722:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765918.503730:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765918.503738:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765918.503752:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076410752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765928.743185:0:36:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743194:0:36:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743205:0:36:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743213:0:36:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765928.743256:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076412032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765928.743302:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743308:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743314:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743320:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743334:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765928.743353:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076412160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743386:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765928.743389:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743394:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743402:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743408:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743432:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765928.743457:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076412288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765928.743486:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743501:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743509:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743515:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743522:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765928.743545:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076412416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743572:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743577:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765928.743578:0:36:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743583:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743597:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743600:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743609:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765928.743623:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076412544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743636:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743639:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765928.743642:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743649:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743658:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743665:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743669:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765928.743693:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076412672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765928.743729:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743738:0:36:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743742:0:36:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743754:0:36:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743757:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765928.743777:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076412800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765928.743796:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743803:0:36:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743808:0:36:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743838:0:36:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743857:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765928.743886:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076412928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765928.743901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765928.743909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765928.743914:0:36:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765928.743922:0:36:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2430 00000100:00080000:2.0:1775765928.743930:0:36:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765928.743938:0:36:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765928.743964:0:36:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765928.744004:0:36:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076413056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765928.935127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765928.935192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765929.959158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765930.983116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765932.007138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765933.031169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765934.055096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765935.079139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765935.911182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765936.103116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765936.935132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765937.127131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765937.959218:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765938.151172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765938.983182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765938.983247:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765939.175235:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765940.007149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765940.199128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765941.031205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765941.223190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765942.055180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765942.247115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765943.079225:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765943.271223:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775765944.103212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765944.103234:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103250:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103269:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103280:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765944.103342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076415488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765944.103396:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103402:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103406:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103438:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.103456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765944.103480:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076415616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765944.103507:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103512:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103519:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103524:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.103531:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765944.103549:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076415744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765944.103589:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103600:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103610:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103619:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.103631:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765944.103660:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076415872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765944.103758:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103781:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103798:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103809:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.103818:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765944.103843:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076416000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.103874:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765944.103880:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.103890:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.103898:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.103910:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.103925:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765944.103949:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076416128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.103998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.104017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765944.104023:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.104075:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.104087:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.104097:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.104145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765944.104175:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076416256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.104223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.104229:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765944.104232:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.104238:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.104246:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.104257:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.104272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765944.104295:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076416384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765944.104328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765944.104343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765944.104348:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765944.104354:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2445 00000100:00080000:2.0:1775765944.104359:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765944.104364:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765944.104371:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765944.104393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076416512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765944.295096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765945.319232:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765946.343194:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775765947.175228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765947.367169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765948.391189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775765949.223144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765949.223161:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223172:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223188:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223196:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765949.223242:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076416640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223283:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223286:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765949.223288:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223291:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223294:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223297:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223307:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765949.223319:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076416768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765949.223347:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223352:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223372:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223375:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223380:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765949.223391:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076416896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765949.223402:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223404:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223407:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223410:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223412:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765949.223422:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076417024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765949.223436:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223439:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223441:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223444:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223446:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765949.223458:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076417152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765949.223484:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223492:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223497:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223501:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765949.223526:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076417280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765949.223542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223549:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223553:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223557:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223560:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765949.223567:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076417408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765949.223585:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223588:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223597:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223601:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223605:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765949.223619:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076417536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765949.223648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765949.223654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765949.223656:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765949.223665:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2450 00000100:00080000:2.0:1775765949.223670:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765949.223673:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765949.223680:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765949.223691:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076417664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765949.415208:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765950.247187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765950.439168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765951.271165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765951.463182:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765952.295125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765952.487162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765953.319157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765953.511135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775765954.343169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765954.343199:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343208:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343220:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343226:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765954.343256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076417792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343279:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765954.343283:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343286:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343288:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343291:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343299:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765954.343334:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076417920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765954.343366:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343369:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343372:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343375:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343379:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765954.343391:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076418048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343412:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765954.343417:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343420:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343432:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343436:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765954.343453:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076418176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343462:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765954.343467:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343470:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343472:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343476:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343480:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765954.343491:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076418304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765954.343514:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343531:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343543:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343550:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343563:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765954.343584:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076418432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765954.343625:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343629:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343631:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343634:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343641:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765954.343658:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076418560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765954.343685:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343708:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343720:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343734:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343757:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765954.343773:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076418688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765954.343814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765954.343818:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765954.343824:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765954.343830:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2456 00000100:00080000:2.0:1775765954.343835:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765954.343839:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765954.343846:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765954.343859:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076418816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765954.535128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765955.559126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765956.583132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765957.607119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765958.631192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765959.655137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765960.679141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765961.703103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765962.727123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765963.751083:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765964.583138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765964.775137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765965.607175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765965.799101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765966.631112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765966.631149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765966.823139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765967.655103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765967.847103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765968.679126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765968.871109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765969.703147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765969.703731:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076422272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765969.895104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765969.895104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765970.727111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765970.727203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765970.919138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765970.919138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765971.751130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765971.943104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765971.943128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765972.775167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765972.775218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775765972.775402:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775765972.775447:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765972.775464:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775513:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775532:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775544:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775559:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775571:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775580:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775596:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775765972.775608:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775765972.967106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765972.967111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765973.799094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765973.991102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765973.991102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765974.823101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765974.823140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765975.015095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765975.015096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765975.847090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765976.039120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765976.039122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765976.871129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765976.871188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765977.063110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765977.063139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775765977.895199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765978.087121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765978.087157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765978.919140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765978.919248:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765979.111156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765979.111158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775765979.943216:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775765979.943241:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943249:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765979.943254:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943263:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943272:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943278:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765979.943310:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076423552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943333:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943335:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765979.943337:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943340:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943343:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943346:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943354:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765979.943364:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076423680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943386:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943389:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765979.943390:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943393:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943409:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943413:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943418:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765979.943432:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076423808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943443:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943446:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765979.943449:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943453:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943458:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943463:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943476:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765979.943503:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076423936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943511:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943514:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765979.943516:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943521:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943527:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943534:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943548:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765979.943560:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076424064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943626:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943638:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765979.943643:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943654:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943659:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943662:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943667:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765979.943680:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076424192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943691:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943693:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765979.943695:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943698:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943701:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943707:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943712:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765979.943738:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076424320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943759:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943768:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765979.943775:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943807:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943833:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943848:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943864:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765979.943882:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076424448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765979.943890:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765979.943901:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765979.943905:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765979.943910:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2481 00000100:00080000:2.0:1775765979.943917:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765979.943920:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765979.943924:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765979.943934:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076424576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765980.135154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765980.135161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765980.967154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775765980.967196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765981.159093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765981.159093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765981.991124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765982.183144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765982.183144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765983.015173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765983.207174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765983.207175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765984.039167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765984.231169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765984.231169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775765985.063105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775765985.063161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765985.255125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775765986.087132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765986.279129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775765987.111143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765987.303123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775765988.135125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775765988.327136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775765989.159123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775765989.351137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775765990.183100:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183109:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765990.183113:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183121:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183131:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183139:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775765990.183180:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076425856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183204:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183206:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765990.183208:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183211:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183215:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183219:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183229:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765990.183243:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076425984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183251:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183253:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765990.183254:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183257:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183273:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183276:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183290:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765990.183301:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076426112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183309:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183311:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765990.183313:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183315:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183317:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183319:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183323:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775765990.183333:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076426240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183362:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183369:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765990.183372:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183379:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183382:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183386:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183391:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775765990.183402:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076426368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183417:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183420:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765990.183422:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183427:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183434:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183437:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183442:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765990.183452:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076426496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183459:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183465:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765990.183468:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183472:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183475:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183478:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183481:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765990.183490:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076426624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183521:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183524:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765990.183526:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183531:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183542:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183547:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183552:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775765990.183562:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076426752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765990.183574:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765990.183577:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765990.183581:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765990.183585:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2491 00000100:00080000:1.0:1775765990.183587:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765990.183589:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775765990.183593:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775765990.183610:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076426880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765990.375132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775765991.207141:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765991.399127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765992.423117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765993.447129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765994.471123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775765995.303170:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303179:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765995.303183:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303192:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2496 00010000:00010000:1.0:1775765995.303198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765995.303200:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303210:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775765995.303245:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076427008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303273:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303277:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765995.303280:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303284:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303288:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303293:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303304:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765995.303325:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076427136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303336:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303339:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765995.303341:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303344:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303354:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303358:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303361:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765995.303386:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076427264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303393:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303395:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765995.303396:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303399:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303403:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303406:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303410:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765995.303420:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076427392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303447:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303450:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765995.303451:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303454:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303456:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303459:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303464:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775765995.303474:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076427520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303485:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303490:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765995.303493:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303507:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303512:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303516:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303523:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765995.303543:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076427648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303560:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303563:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765995.303565:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303569:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303574:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303580:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303585:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765995.303600:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076427776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303621:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303624:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765995.303626:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303628:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303630:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303646:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303649:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765995.303657:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076427904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765995.303690:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765995.303694:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765995.303698:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765995.303705:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2496 00000100:00080000:2.0:1775765995.303712:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765995.303717:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775765995.303724:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775765995.303734:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076428032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775765995.495187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775765996.519136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775765997.351248:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765997.543113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775765998.375208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765998.567093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775765999.399217:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765999.591143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766000.423137:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423149:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766000.423153:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423163:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423174:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423182:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766000.423217:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076428160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423243:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423248:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766000.423265:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423275:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423284:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423290:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423301:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766000.423314:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076428288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423325:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423328:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766000.423330:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423334:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423338:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423341:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423345:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766000.423357:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076428416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423364:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423366:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766000.423368:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423370:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423373:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423375:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423379:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766000.423393:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076428544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423402:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423404:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766000.423406:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423409:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423411:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423413:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423416:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766000.423421:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076428672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423427:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423429:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766000.423430:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423432:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423433:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423435:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423438:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766000.423442:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076428800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423448:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423450:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766000.423451:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423458:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423463:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423472:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423478:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766000.423493:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076428928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423508:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423515:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766000.423519:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423528:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423532:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423537:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423541:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766000.423554:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076429056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766000.423567:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766000.423571:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766000.423573:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766000.423578:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2502 00000100:00080000:2.0:1775766000.423608:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766000.423616:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766000.423636:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766000.423654:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076429184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766000.615135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766001.447169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766001.639141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766002.663108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766003.495169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766003.687122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766004.519134:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766004.711153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766005.543131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766005.543159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766005.735233:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766006.567102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766006.567207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766006.759093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766007.783229:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766008.807147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766009.831123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766010.663172:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663180:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766010.663221:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663233:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663246:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663254:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766010.663301:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076430464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663331:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663336:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766010.663337:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663342:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663345:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663350:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663370:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766010.663385:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076430592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663410:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663414:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766010.663417:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663424:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663427:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663430:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663436:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766010.663446:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076430720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663512:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663518:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766010.663520:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663530:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663540:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663555:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663567:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766010.663602:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076430848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663630:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663633:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766010.663641:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663652:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663659:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663665:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663673:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766010.663702:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076430976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663722:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663726:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766010.663729:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663737:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663762:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663778:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663790:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766010.663820:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076431104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663861:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663872:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766010.663878:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.663897:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.663917:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.663933:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.663946:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766010.663969:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076431232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.663982:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.663993:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766010.663997:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.664005:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.664014:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.664030:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.664062:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766010.664083:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076431360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766010.664140:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766010.664143:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766010.664146:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766010.664153:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2512 00000100:00080000:1.0:1775766010.664164:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766010.664171:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766010.664179:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766010.664207:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076431488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766010.855122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766011.879114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766012.903149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766013.927171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766014.951154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766015.783206:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783214:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766015.783220:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783230:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783242:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783251:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766015.783287:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076431616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783314:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783317:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766015.783333:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783337:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783342:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783346:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783355:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766015.783369:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076431744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783385:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783389:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766015.783391:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783396:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783401:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783406:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783412:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766015.783427:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076431872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783471:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783474:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766015.783476:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783483:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783488:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783497:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783509:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766015.783536:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076432000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783557:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783563:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766015.783567:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783573:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783577:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783581:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783585:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766015.783599:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076432128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783610:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783613:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766015.783616:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783623:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783628:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783632:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783637:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766015.783653:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076432256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783678:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783682:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766015.783686:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783703:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783711:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783715:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783720:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766015.783735:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076432384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783751:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783759:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766015.783761:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783768:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783774:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783780:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783783:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766015.783793:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076432512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766015.783808:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766015.783813:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766015.783817:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766015.783824:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2517 00000100:00080000:2.0:1775766015.783827:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766015.783832:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766015.783840:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766015.783849:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076432640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766015.975105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766016.999123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766018.023125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766019.047131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766020.071105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766020.903177:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903184:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766020.903189:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903197:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903209:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903214:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766020.903249:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076432768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903272:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903275:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766020.903276:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903279:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903282:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903287:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903295:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766020.903305:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076432896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903313:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903319:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766020.903338:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903343:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903347:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903350:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903354:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766020.903366:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076433024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903373:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903375:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766020.903376:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903379:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903407:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903411:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903415:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766020.903424:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076433152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903442:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903445:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766020.903446:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903449:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903452:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903454:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903458:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766020.903474:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076433280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903481:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903484:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766020.903486:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903490:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903494:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903504:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903514:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766020.903538:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076433408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903580:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903590:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766020.903597:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903608:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903614:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903618:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903624:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766020.903639:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076433536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903650:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903654:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766020.903655:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903669:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903674:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903684:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903702:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766020.903716:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076433664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766020.903759:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766020.903763:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766020.903766:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766020.903775:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2522 00000100:00080000:2.0:1775766020.903787:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766020.903796:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766020.903802:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766020.903816:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076433792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766021.095148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766021.927189:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766022.119110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766022.951163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766023.143126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766023.975111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766024.167098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766024.999186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766025.191182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766026.023101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766026.023735:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766026.215127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766027.047123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766027.239162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766028.071107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766028.263098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766029.095125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766029.287177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766030.119115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766030.311152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766031.143144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766031.143770:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766031.335156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766032.167100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766032.359138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766033.191186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766033.383126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766034.215112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766034.407194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766035.239096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766035.431147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766036.263124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766036.263289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076436224:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766036.263439:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076436352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766036.263498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076436480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766036.263553:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076436608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766036.263591:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076436736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766036.263648:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076436864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766036.264997:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076436992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766036.265177:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076437120:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775766036.265424:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766036.455105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766037.287223:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766037.479181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766038.311211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766038.311312:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766038.503215:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766039.335174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:1.0:1775766039.335504:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775766039.335592:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766039.335626:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335734:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335765:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335787:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335807:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335827:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335857:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335898:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766039.335967:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766039.527133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766040.359237:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766040.551109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766041.383138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766041.384321:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766041.575112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766042.407130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766042.599131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766043.431129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766043.431217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766043.623103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766044.455123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766044.647109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766045.479195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766045.479269:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766045.671118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766046.503183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766046.503913:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766046.695130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766047.527091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766047.527132:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766047.719113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766048.551177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766048.743148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766049.575182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766049.767238:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766050.599101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766050.599166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766050.791178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766051.623136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766051.623186:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623192:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766051.623196:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623205:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623215:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623223:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766051.623262:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076439680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623284:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623287:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766051.623298:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623302:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623307:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623310:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623319:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766051.623331:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076439808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623340:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623342:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766051.623343:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623346:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623349:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623351:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623354:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766051.623362:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076439936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623368:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623371:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766051.623372:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623380:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623384:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623387:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623391:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766051.623404:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076440064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623411:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623415:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766051.623418:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623426:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623432:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623436:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623439:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766051.623449:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076440192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623464:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623467:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766051.623471:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623476:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623478:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623481:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623483:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766051.623489:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076440320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623496:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623501:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766051.623505:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623515:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623521:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623527:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623533:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766051.623548:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076440448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623571:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623582:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766051.623587:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623595:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623603:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623609:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623617:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766051.623630:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076440576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766051.623644:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766051.623646:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766051.623649:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766051.623666:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2553 00000100:00080000:1.0:1775766051.623673:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766051.623678:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766051.623684:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766051.623699:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076440704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766051.815181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766052.647152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766052.839159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766053.671190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766053.863131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766054.695155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766054.887128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766055.719149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766055.911139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766056.743143:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743155:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766056.743160:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743172:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2558 00010000:00010000:2.0:1775766056.743183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766056.743185:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743194:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766056.743233:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076440832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743264:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743268:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766056.743270:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743275:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743279:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743284:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743295:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766056.743313:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076440960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743326:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743330:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766056.743345:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743350:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743356:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743364:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743371:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766056.743386:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076441088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743414:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743419:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766056.743424:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743433:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743439:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743446:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743455:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766056.743472:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076441216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743491:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743495:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766056.743500:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743508:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743515:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743524:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743531:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766056.743562:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076441344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743580:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743587:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766056.743591:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743601:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743612:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743619:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743628:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766056.743647:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076441472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743665:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743669:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766056.743673:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743683:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743690:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743697:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743705:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766056.743734:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076441600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743752:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743762:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766056.743767:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743784:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743793:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743800:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743811:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766056.743832:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076441728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766056.743858:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766056.743868:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766056.743871:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766056.743881:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2558 00000100:00080000:1.0:1775766056.743890:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766056.743899:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766056.743905:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766056.743924:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076441856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766056.935131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766057.959177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766058.791346:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766058.983095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766060.007170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766060.839203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766061.031134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766061.863116:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863124:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766061.863128:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863138:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863149:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863156:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766061.863193:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076441984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863213:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863228:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766061.863230:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863233:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863237:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863243:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863253:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766061.863266:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076442112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863275:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863278:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766061.863279:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863282:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863291:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863294:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863298:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766061.863309:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076442240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863317:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863319:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766061.863321:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863323:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863326:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863328:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863339:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766061.863349:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076442368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863379:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863381:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766061.863382:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863385:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863390:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863395:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863401:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766061.863409:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076442496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863420:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863426:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766061.863428:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863435:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863438:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863442:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863444:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766061.863450:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076442624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863467:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863470:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766061.863476:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863480:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863483:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863489:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863497:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766061.863508:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076442752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863516:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863521:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766061.863524:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863529:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863532:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863537:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863544:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766061.863559:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076442880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766061.863581:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766061.863586:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766061.863588:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766061.863593:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2563 00000100:00080000:1.0:1775766061.863600:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766061.863606:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766061.863612:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766061.863627:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076443008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766062.055127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766062.887268:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766063.079169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766064.103158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766064.935226:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766064.935237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766065.127126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766065.959165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766066.151196:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766066.983104:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766066.983107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766066.983115:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766066.983119:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983127:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983138:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983143:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766066.983177:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076443136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983197:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983200:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766066.983202:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983205:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983208:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983211:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983219:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766066.983230:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076443264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983238:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983241:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766066.983248:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983252:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983255:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983258:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983262:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766066.983275:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076443392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983284:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983287:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766066.983289:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983291:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983294:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983296:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983300:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766066.983307:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076443520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983313:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983316:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766066.983319:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983326:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983332:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983336:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983340:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766066.983358:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076443648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983374:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983383:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766066.983385:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983388:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983391:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983395:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983397:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766066.983406:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076443776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983416:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983418:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766066.983420:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983422:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983425:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983431:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983435:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766066.983445:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076443904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983467:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983472:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766066.983475:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983484:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983489:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983496:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983505:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766066.983515:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076444032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766066.983522:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766066.983528:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766066.983530:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766066.983535:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2568 00000100:00080000:2.0:1775766066.983539:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766066.983545:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766066.983552:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766066.983561:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076444160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766067.175114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766068.199160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766069.031248:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766069.223131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766070.247192:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766071.079273:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766071.271194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766072.295180:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766073.127211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766073.319167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766074.343107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766075.175245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766075.367134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766076.391101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766077.223179:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223187:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766077.223192:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223204:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223217:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223225:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766077.223265:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076445440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223295:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223299:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766077.223301:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223305:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223308:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223311:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223321:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766077.223334:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076445568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223344:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223347:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766077.223360:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223368:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223376:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223381:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223387:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766077.223401:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076445696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223412:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223416:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766077.223418:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223421:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223425:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223428:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223433:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766077.223447:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076445824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223459:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223462:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766077.223465:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223469:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223472:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223475:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223480:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766077.223501:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076445952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223526:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223530:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766077.223535:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223546:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223554:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223559:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223564:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766077.223594:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076446080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223615:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223624:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766077.223626:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223632:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223635:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223640:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223647:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766077.223675:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076446208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223727:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223741:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766077.223747:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223771:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223782:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223787:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223795:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766077.223817:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076446336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766077.223838:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766077.223844:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766077.223847:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766077.223876:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2578 00000100:00080000:1.0:1775766077.223886:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766077.223905:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766077.223920:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766077.223946:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076446464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766077.415156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766078.439103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766079.463183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766080.487081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766081.511142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766082.343213:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343230:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766082.343238:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343271:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343295:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343306:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766082.343370:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076446592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.343412:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343442:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766082.343445:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343455:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343470:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343489:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.343504:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766082.343525:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076446720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.343556:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343561:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766082.343563:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343570:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343578:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343582:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.343588:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766082.343625:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076446848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.343641:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343647:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766082.343650:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343658:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343665:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343671:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.343700:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766082.343718:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076446976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.343773:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343778:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766082.343785:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343793:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343841:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343850:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.343867:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766082.343897:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076447104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.343926:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.343941:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766082.343948:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.343956:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.343964:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.343971:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.343979:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766082.344011:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076447232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.344125:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.344131:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766082.344149:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.344169:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.344178:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.344184:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.344194:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766082.344220:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076447360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.344268:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.344278:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766082.344286:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.344301:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.344312:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.344321:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.344341:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766082.344362:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076447488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766082.344377:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766082.344382:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766082.344385:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766082.344399:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2584 00000100:00080000:1.0:1775766082.344409:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766082.344419:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766082.344424:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766082.344452:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076447616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766082.535192:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766083.559114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766084.583171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766085.607111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766086.631132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766087.463191:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463199:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766087.463204:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463217:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463243:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463251:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766087.463289:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076447744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463315:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463318:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766087.463319:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463322:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463326:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463330:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463339:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766087.463352:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076447872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463361:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463364:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766087.463364:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463367:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463369:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463383:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463387:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766087.463395:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076448000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463402:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463405:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766087.463408:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463415:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463419:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463422:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463426:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766087.463435:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076448128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463441:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463443:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766087.463445:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463451:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463456:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463462:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463467:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766087.463477:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076448256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463491:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463503:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766087.463506:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463514:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463537:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463541:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463546:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766087.463555:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076448384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463576:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463579:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766087.463581:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463585:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463590:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463597:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463603:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766087.463616:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076448512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463628:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463630:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766087.463632:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463636:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463642:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463648:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463660:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766087.463679:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076448640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766087.463696:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766087.463700:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766087.463702:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766087.463709:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2589 00000100:00080000:1.0:1775766087.463716:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766087.463720:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766087.463724:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766087.463735:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076448768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766087.655112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766088.679137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766089.703113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766090.727099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766091.751153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766092.583159:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583167:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766092.583171:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583194:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583204:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583213:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766092.583249:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076448896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583270:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583274:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766092.583275:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583278:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583290:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583294:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583303:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766092.583316:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076449024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583325:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583327:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766092.583328:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583332:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583335:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583337:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583340:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766092.583348:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076449152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583354:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583356:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766092.583357:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583363:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583367:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583370:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583374:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766092.583382:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076449280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583397:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583400:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766092.583401:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583407:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583413:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583418:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583422:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766092.583433:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076449408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583442:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583446:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766092.583449:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583455:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583460:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583464:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583471:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766092.583481:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076449536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583489:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583491:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766092.583493:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583495:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583504:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583509:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583513:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766092.583523:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076449664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583537:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583540:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766092.583543:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583549:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583554:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583558:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583562:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766092.583572:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076449792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766092.583588:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766092.583591:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766092.583594:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766092.583601:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2594 00000100:00080000:1.0:1775766092.583605:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766092.583607:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766092.583611:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766092.583621:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076449920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766092.775163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766093.607196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766093.799151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766094.631203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766094.823146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766095.655163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766095.847148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766096.679178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766096.679250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766096.871206:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766097.703172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766097.895135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766098.727219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766098.727254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766098.919120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766099.751183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766099.943185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766100.775174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766100.967110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766101.799146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766101.991163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766102.823195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766103.015185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766103.847201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766104.039094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766104.871311:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766105.063167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766105.895251:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766106.087163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766106.919224:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766106.919246:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766107.111114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766107.943103:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943114:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766107.943118:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943128:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943138:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943146:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766107.943181:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076452352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943215:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943220:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766107.943222:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943228:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943233:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943238:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943248:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766107.943264:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076452480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943275:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943279:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766107.943281:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943285:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943289:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943293:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943307:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766107.943324:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076452608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943335:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943341:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766107.943344:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943350:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943356:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943362:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943368:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766107.943382:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076452736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943395:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943404:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766107.943408:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943416:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943421:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943444:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943453:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766107.943469:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076452864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943481:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943484:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766107.943494:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943498:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943504:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943513:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943520:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766107.943537:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076452992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943549:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943554:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766107.943566:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943572:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943577:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943581:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943587:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766107.943602:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076453120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943612:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943617:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766107.943620:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943629:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943636:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943643:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943650:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766107.943672:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076453248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766107.943684:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766107.943687:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766107.943689:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766107.943694:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2609 00000100:00080000:2.0:1775766107.943699:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766107.943706:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766107.943712:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766107.943729:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076453376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766108.135105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766108.967204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766109.159125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766110.183142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766111.015166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766111.207081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766112.231116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766113.063171:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063178:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766113.063183:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063194:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063205:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063236:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766113.063292:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076453504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063367:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063370:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766113.063371:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063375:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063378:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063382:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063391:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766113.063439:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076453632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063462:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063470:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766113.063474:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063480:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063491:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063499:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063545:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766113.063580:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076453760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063588:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063590:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766113.063592:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063595:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063607:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063612:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063616:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766113.063631:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076453888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063658:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063665:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766113.063669:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063679:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063683:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063687:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063692:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766113.063706:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076454016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063715:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063721:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766113.063723:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063727:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063732:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063740:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063749:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766113.063789:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076454144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063860:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063891:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766113.063895:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063903:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063909:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063916:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063920:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766113.063935:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076454272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063953:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.063959:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766113.063960:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.063964:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.063966:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.063969:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.063972:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766113.063984:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076454400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766113.063997:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766113.064005:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766113.064008:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766113.064013:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2614 00000100:00080000:1.0:1775766113.064026:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766113.064034:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766113.064052:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766113.064062:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076454528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766113.255084:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766114.279116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766115.303187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766116.327181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766117.351163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766118.183170:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183182:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766118.183188:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183202:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183217:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183228:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766118.183280:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076454656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183312:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183316:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766118.183318:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183322:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183325:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183330:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183343:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766118.183373:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076454784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183386:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183389:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766118.183391:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183395:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183400:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183403:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183408:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766118.183419:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076454912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183429:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183478:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766118.183484:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183492:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183514:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183520:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183527:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766118.183541:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076455040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183551:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183555:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766118.183557:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183587:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183592:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183598:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183610:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766118.183628:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076455168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183652:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183656:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766118.183658:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183663:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183666:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183670:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183675:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766118.183691:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076455296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183703:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183724:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766118.183725:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183736:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183743:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183748:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183754:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766118.183770:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076455424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183845:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183856:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766118.183861:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183874:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183882:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183890:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.183902:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766118.183927:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076455552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766118.183956:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766118.183963:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766118.183968:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766118.183981:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2619 00000100:00080000:1.0:1775766118.183986:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766118.183996:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766118.184004:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766118.184018:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076455680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766118.184145:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076455168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766118.184161:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766118.184164:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766118.184167:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076455168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766118.184189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766118.184208:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766118.184218:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766118/real 1775766118] req@ffffa04350932680 x1862023076455296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766118.184228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076455296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766118.184236:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766118.184239:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766118.184241:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766118.184246:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766118.184259:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766118.184266:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766118/real 1775766118] req@ffffa04350930a80 x1862023076455424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766118.184275:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076455424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766118.184282:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766118.184285:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766118.184294:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766118.184299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766118.184317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766118.184325:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766118/real 1775766118] req@ffffa04350930000 x1862023076455552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766118.184336:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076455552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766118.184342:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766118.184345:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766118.184347:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766118.184351:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766118.184365:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766118.184372:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766118/real 1775766118] req@ffffa04350930e00 x1862023076455680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766118.184383:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076455680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766118.184389:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766118.184392:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766118.184394:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076455680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766118.375187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766118.375187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766119.207157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766119.207184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766119.399113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766119.399161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766120.231121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766120.423200:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766120.423203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766121.255156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766121.447120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766121.447121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766122.279205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766122.471170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766122.471170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766123.303118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766123.303181:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303188:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766123.303192:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303202:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303213:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303219:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766123.303277:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076455808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766123.303303:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303307:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766123.303308:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303332:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00100000:0.0:1775766123.303334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076455808:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766123.303337:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303343:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303353:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766123.303366:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076455936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766123.303374:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303377:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766123.303378:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303382:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303384:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303387:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303390:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766123.303397:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076456064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766123.303404:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303428:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766123.303431:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766123.303434:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766123.303438:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303442:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303446:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303451:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766123.303472:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076456192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766123.303485:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303488:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766123.303489:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303493:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303497:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766123.303498:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7e680 x1862023076455808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766123.303502:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303508:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766123.303520:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076456320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766123.303524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076455808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766123.303537:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303542:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766123.303544:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766123.303544:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303553:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:0.0:1775766123.303555:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303558:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076455808:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766123.303564:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303569:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303572:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766123.303580:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076456448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766123.303584:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076455936:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766123.303597:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303600:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766123.303602:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303605:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2624 00000800:00000100:0.0:1775766123.303609:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766123.303616:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303621:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7df80 x1862023076455936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766123.303627:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766123.303639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076455936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766123.303642:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076456576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766123.303662:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766123.303664:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766123.303668:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766123.303669:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766123.303671:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303674:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303676:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766123.303682:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076455936:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766123.303686:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766123.303692:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766123.303697:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076456704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766123.303703:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766123.303706:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766123.303707:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766123.303710:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2624 00000100:00080000:2.0:1775766123.303713:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766123.303716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766123.303719:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766123.303724:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766123.303729:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7d500 x1862023076456064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766123.303734:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076456832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766123.303751:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076456064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303781:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766123.303793:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766123.303807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303813:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7f480 x1862023076456192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.303820:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076456192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766123.303833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766123.303842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7d880 x1862023076456320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.303863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076456320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076456320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766123.303887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766123.303898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7c380 x1862023076456448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.303912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076456448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766123.303924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766123.303933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7f100 x1862023076456576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.303945:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076456576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766123.303962:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766123.303975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.303981:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7ea00 x1862023076456704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.303988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076456704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.303993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.303995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.303996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766123.303999:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766123.304008:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766123.304014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766123/real 1775766123] req@ffffa0434aa7e680 x1862023076456832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766123.304021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076456832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766123.304045:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766123.304047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766123.304054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076456832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766123.495145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766123.495145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766124.327171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766124.519141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766124.519142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766125.351157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766125.543142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766125.543148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766126.375125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766126.567100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766126.567112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766127.399142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766127.591128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766127.591160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766128.423117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766128.423149:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766128.423159:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766128.423165:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423177:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423188:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423195:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766128.423242:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076456960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766128.423269:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766128.423273:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766128.423274:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423277:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423280:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423284:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423293:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766128.423304:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076457088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766128.423314:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076456960:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766128.423315:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766128.423328:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766128.423332:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423358:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423397:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423404:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766128.423406:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423420:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766128.423439:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076457216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766128.423450:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766128.423455:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350930e00 x1862023076456960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766128.423459:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766128.423465:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423481:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423490:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423494:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423498:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766128.423500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076456960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.423512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766128.423512:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076457344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766128.423529:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766128.423545:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766128.423546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076456960:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766128.423554:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766128.423560:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423565:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423568:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423571:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423574:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766128.423583:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766128.423588:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076457472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766128.423605:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766128.423614:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766128.423617:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766128.423619:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423628:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423632:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423648:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766128.423654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350930000 x1862023076457088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766128.423658:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766128.423680:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076457600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766128.423684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076457088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766128.423691:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766128.423694:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766128.423698:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423701:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423714:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766128.423726:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423741:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766128.423747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766128.423752:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076457728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766128.423754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766128.423771:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766128.423772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457088:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766128.423775:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766128.423778:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423792:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423802:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766128.423804:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766128.423811:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423820:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766128.423834:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076457856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766128.423841:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766128.423855:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766128.423857:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766128.423860:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2630 00000100:00080000:1.0:1775766128.423863:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766128.423872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766128.423876:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766128.423881:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766128.423893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350930a80 x1862023076457216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766128.423894:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076457984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766128.423976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076457216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.423984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.423990:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.423993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766128.423998:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766128.424013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350932680 x1862023076457344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424027:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076457344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424032:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424034:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424043:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766128.424059:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766128.424094:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424103:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350931880 x1862023076457472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076457472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424118:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424120:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424138:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076457472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766128.424146:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766128.424170:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424184:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350930700 x1862023076457600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424199:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076457600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424210:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766128.424225:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766128.424243:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424260:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350931c00 x1862023076457728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424270:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076457728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424277:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424279:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424282:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766128.424286:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766128.424300:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424307:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350932a00 x1862023076457856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076457856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424321:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424324:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424326:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766128.424330:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766128.424357:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766128.424377:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766128/real 1775766128] req@ffffa04350932300 x1862023076457984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766128.424385:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076457984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766128.424395:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766128.424398:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766128.424400:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076457984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766128.615152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766128.615202:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766129.447137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766129.447203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766129.639174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766129.639174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766130.471132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766130.471183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766130.663121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766130.663149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766131.495140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766131.495216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766131.687128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766131.687128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766132.519134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766132.519196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766132.711116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766132.711133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766133.543129:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766133.543134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766133.543148:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766133.543152:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766133.543164:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543204:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543212:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766133.543260:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076458112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766133.543289:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543293:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766133.543296:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766133.543300:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543305:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543310:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543335:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766133.543338:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076458112:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766133.543366:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076458240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766133.543379:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543383:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766133.543385:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766133.543389:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543394:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543399:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543404:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766133.543409:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766133.543420:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076458368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766133.543431:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543441:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766133.543443:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766133.543445:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350932300 x1862023076458112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766133.543452:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543461:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766133.543465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076458112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766133.543468:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543480:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766133.543481:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.543490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076458112:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766133.543499:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076458496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766133.543511:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458240:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766133.543514:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543518:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766133.543522:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766133.543528:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543533:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543537:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766133.543542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543543:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766133.543554:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350932a00 x1862023076458240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766133.543561:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076458624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766133.543569:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076458240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766133.543582:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766133.543583:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543588:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766133.543590:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775766133.543593:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458240:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766133.543595:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766133.543599:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766133.543602:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543608:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543613:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766133.543616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766133.543623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350931c00 x1862023076458368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766133.543632:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766133.543634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076458368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766133.543643:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.543650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766133.543652:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076458752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766133.543664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458496:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766133.543667:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543672:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766133.543675:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766133.543680:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766133.543683:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00100000:0.0:1775766133.543689:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350930700 x1862023076458496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766133.543689:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543698:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543704:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766133.543719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076458496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766133.543723:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076458880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766133.543730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.543744:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458496:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766133.543748:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543753:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766133.543754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458624:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766133.543755:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766133.543764:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543769:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543773:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766133.543792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766133.543799:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766133.543806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350931880 x1862023076458624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766133.543815:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076459008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766133.543824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076458624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766133.543834:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766133.543840:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766133.543841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543846:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766133.543850:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766133.543853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076458624:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766133.543871:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2635 00000100:00080000:1.0:1775766133.543877:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766133.543883:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766133.543885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076458752:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766133.543892:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775766133.543911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766133.543911:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076459136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766133.543920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350932680 x1862023076458752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766133.543933:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076458752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766133.543948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543952:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.543954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076458752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766133.543958:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076458880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766133.543971:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766133.543976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350930a80 x1862023076458880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766133.543984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076458880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766133.543989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.543991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.543993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076458880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766133.543996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766133.544005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766133.544010:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350930000 x1862023076459008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766133.544016:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076459008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766133.544021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.544023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.544046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766133.544059:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766133.544073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766133.544081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766133/real 1775766133] req@ffffa04350930e00 x1862023076459136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766133.544088:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076459136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766133.544093:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766133.544095:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766133.544097:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766133.735083:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766133.735088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766134.567113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766134.567169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766134.759113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766134.759113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766135.591128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766135.783166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766135.783234:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766136.615155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766136.615213:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766136.807103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766136.807164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766137.639135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766137.639172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766137.831122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766137.831122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766138.663127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766138.663137:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766138.663145:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766138.663150:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766138.663160:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:1.0:1775766138.663172:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766138.663179:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766138.663221:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076459264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766138.663248:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766138.663252:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775766138.663282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076459264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766138.663293:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766138.663300:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:1.0:1775766138.663306:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766138.663311:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663322:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766138.663338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766138.663342:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076459392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766138.663356:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766138.663360:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766138.663362:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766138.663367:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:1.0:1775766138.663372:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766138.663376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350930e00 x1862023076459264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663381:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663392:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766138.663398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076459264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766138.663407:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076459520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766138.663442:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766138.663442:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766138.663448:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766138.663448:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766138.663452:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766138.663455:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076459264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766138.663460:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:1.0:1775766138.663465:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766138.663470:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663476:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766138.663479:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766138.663502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766138.663506:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076459648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766138.663513:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350930000 x1862023076459392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663528:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766138.663531:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076459392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663548:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766138.663553:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766138.663554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.663562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766138.663563:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00100000:0.0:1775766138.663566:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766138.663572:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766138.663572:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766138.663581:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663588:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775766138.663590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663600:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350930a80 x1862023076459520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766138.663608:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076459776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766138.663612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076459520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766138.663636:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.663641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766138.663643:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766138.663646:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766138.663648:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766138.663651:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766138.663661:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:1.0:1775766138.663666:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766138.663670:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459648:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766138.663673:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663680:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766138.663692:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766138.663698:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076459904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766138.663702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350932680 x1862023076459648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663716:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766138.663726:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766138.663730:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766138.663734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076459648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766138.663745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766138.663747:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:0.0:1775766138.663750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766138.663755:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459648:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766138.663756:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766138.663760:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459776:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766138.663765:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663773:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766138.663779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350931880 x1862023076459776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766138.663792:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076460032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766138.663824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076459776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663829:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766138.663836:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766138.663841:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766138.663846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766138.663848:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00080000:0.0:1775766138.663852:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766138.663854:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766138.663856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076459776:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766138.663862:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663869:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459904:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766138.663869:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766138.663887:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076460160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766138.663891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663898:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350930700 x1862023076459904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663928:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766138.663930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076459904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766138.663936:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766138.663941:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766138.663943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.663949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766138.663949:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2640 00000100:00100000:0.0:1775766138.663952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076459904:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766138.663957:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766138.663959:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766138.663965:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766138.663972:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775766138.663978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.663989:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350931c00 x1862023076460032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766138.663989:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076460288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766138.663998:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076460032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766138.664023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.664060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766138.664064:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766138.664079:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766138.664093:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.664099:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350932a00 x1862023076460160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766138.664106:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076460160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766138.664121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.664123:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766138.664125:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766138.664129:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766138.664142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766138.664148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766138/real 1775766138] req@ffffa04350932300 x1862023076460288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766138.664155:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076460288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766138.664160:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766138.664162:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766138.664164:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076460288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766138.855097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766138.855101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766139.687141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766139.687219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766139.879130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766139.879131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766140.711100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766140.711135:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766140.903125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766140.903125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766141.735127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766141.735187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766141.927112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766141.927126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766142.759115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766142.759194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766142.951121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766142.951147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766143.783142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766143.783161:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783168:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766143.783172:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783181:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783192:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766143.783199:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766143.783239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766143.783251:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076460416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783274:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783277:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766143.783279:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783283:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783287:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766143.783292:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783300:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766143.783312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076460416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766143.783317:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076460544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783340:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783343:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766143.783344:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783348:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783352:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766143.783356:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783361:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766143.783372:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076460672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783380:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783384:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766143.783386:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766143.783388:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766143.783393:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783398:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766143.783402:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783407:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766143.783418:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076460800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783428:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766143.783432:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350932300 x1862023076460416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783433:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766143.783437:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783448:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783458:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766143.783459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076460416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783467:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766143.783477:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766143.783477:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775766143.783486:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.783492:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076460416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766143.783494:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076460928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783504:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783506:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766143.783509:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783512:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783516:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766143.783522:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460544:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766143.783522:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783531:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766143.783552:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076461056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766143.783553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766143.783565:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766143.783570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350932a00 x1862023076460544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783570:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766143.783574:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783584:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783591:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766143.783595:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076460544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783598:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766143.783613:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.783619:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775766143.783619:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766143.783624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766143.783642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766143.783642:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076461184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766143.783666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766143.783677:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766143.783678:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350931c00 x1862023076460672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783694:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766143.783698:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783707:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783715:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766143.783719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076460672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766143.783721:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783731:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766143.783732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.783740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.783743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766143.783761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766143.783761:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076461312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766143.783776:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766143.783779:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766143.783781:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766143.783785:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2645 00000100:00080000:1.0:1775766143.783788:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766143.783792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766143.783796:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766143.783801:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766143.783806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350930700 x1862023076460800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766143.783812:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076461440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766143.783824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076460800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.783832:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.783835:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.783838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766143.783843:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766143.783867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766143.783879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350931880 x1862023076460928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766143.783893:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076460928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.783918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.783923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.783928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076460928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766143.783952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766143.783980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766143.783996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350932680 x1862023076461056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766143.784014:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076461056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.784060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.784067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.784072:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766143.784081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766143.784109:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766143.784127:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350930a80 x1862023076461184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766143.784141:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076461184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.784150:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.784154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.784165:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766143.784172:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766143.784197:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766143.784209:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350930000 x1862023076461312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766143.784234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076461312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.784241:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.784243:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.784245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766143.784250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766143.784265:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766143.784272:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766143/real 1775766143] req@ffffa04350930e00 x1862023076461440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766143.784280:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076461440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766143.784286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766143.784288:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766143.784290:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076461440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766143.975129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766143.975131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766144.807128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766144.807138:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766144.999122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766144.999132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766145.831113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766145.831176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766146.023125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766146.023133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766146.855111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766146.855172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766147.047131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766147.047145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766147.879146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766147.879263:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766148.071150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766148.071154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766148.903144:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766148.903145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766148.903156:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766148.903162:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903189:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903201:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903209:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766148.903251:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076461568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766148.903276:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903281:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766148.903284:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903289:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903294:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903300:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766148.903312:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766148.903318:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076461568:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766148.903326:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076461696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766148.903339:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903342:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766148.903344:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903348:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903381:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903386:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766148.903390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766148.903393:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766148.903406:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076461824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766148.903417:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903421:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766148.903423:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903428:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00100000:0.0:1775766148.903430:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350930e00 x1862023076461568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766148.903432:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903437:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766148.903442:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766148.903451:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076461568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766148.903453:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076461952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766148.903465:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766148.903465:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903469:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766148.903470:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766148.903471:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766148.903475:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076461568:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766148.903476:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903480:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903483:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766148.903488:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766148.903498:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076462080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766148.903501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766148.903527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766148.903528:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903532:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766148.903534:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903539:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00100000:0.0:1775766148.903540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350930000 x1862023076461696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766148.903543:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903549:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903555:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076461696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766148.903555:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766148.903567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766148.903567:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076462208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766148.903571:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903576:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461696:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766148.903578:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903582:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766148.903583:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461824:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766148.903584:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903589:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903593:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903597:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766148.903602:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766148.903613:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076462336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766148.903615:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766148.903625:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766148.903628:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350930a80 x1862023076461824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766148.903629:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766148.903631:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766148.903636:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00100000:0.0:1775766148.903643:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076461824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766148.903646:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903651:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766148.903654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766148.903657:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766148.903658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766148.903668:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076462464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766148.903669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461952:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766148.903680:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766148.903684:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766148.903686:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766148.903689:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766148.903691:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2650 00000100:00080000:1.0:1775766148.903695:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766148.903699:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903701:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350932680 x1862023076461952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766148.903704:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766148.903716:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076461952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766148.903716:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076462592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766148.903728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.903744:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903748:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076461952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766148.903771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766148.903794:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903804:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350931880 x1862023076462080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766148.903827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076462080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766148.903834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.903837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766148.903853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766148.903871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350930700 x1862023076462208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766148.903888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076462208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766148.903895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.903897:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766148.903903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766148.903915:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350931c00 x1862023076462336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766148.903930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076462336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766148.903947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.903950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766148.903957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766148.903971:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766148.903979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350932a00 x1862023076462464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766148.903988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076462464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766148.903994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.903996:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.903998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766148.904002:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766148.904014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766148.904020:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766148/real 1775766148] req@ffffa04350932300 x1862023076462592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766148.904050:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076462592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766148.904056:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766148.904059:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766148.904084:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076462592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766149.095152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766149.095158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766149.927100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766149.927150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766150.119136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766150.119149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766150.951095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766151.143186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766151.143203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766151.975149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766151.975159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766152.167119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766152.167132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766152.999178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766152.999198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766153.191157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766153.191161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766154.023171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766154.023192:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766154.023200:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766154.023204:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766154.023214:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023223:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023232:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766154.023270:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076462720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766154.023303:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1775766154.023308:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766154.023309:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766154.023314:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766154.023322:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023326:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766154.023340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076462720:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766154.023348:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023360:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766154.023376:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076462848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766154.023388:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766154.023392:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766154.023394:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766154.023398:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766154.023402:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023408:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023413:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023418:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766154.023429:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076462976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766154.023439:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766154.023442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7ed80 x1862023076462720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766154.023443:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766154.023445:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766154.023449:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023454:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023463:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023470:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766154.023485:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076463104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766154.023486:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076462720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766154.023509:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766154.023513:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766154.023521:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766154.023522:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766154.023525:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766154.023530:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076462720:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766154.023534:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023540:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023545:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023550:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766154.023560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766154.023572:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076463232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766154.023583:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766154.023587:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766154.023589:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766154.023594:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766154.023595:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023601:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023606:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023611:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766154.023612:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7fb80 x1862023076462848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.023636:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076462848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766154.023639:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076463360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766154.023653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766154.023659:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766154.023660:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766154.023666:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766154.023667:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766154.023669:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766154.023676:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00100000:1.0:1775766154.023678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766154.023681:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023686:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766154.023697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023697:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766154.023707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7ce00 x1862023076462976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766154.023714:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076463488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766154.023721:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076462976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766154.023730:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766154.023734:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766154.023736:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766154.023741:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023747:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766154.023754:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766154.023763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766154.023765:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766154.023768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.023774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076462976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766154.023779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076463104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766154.023788:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076463616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766154.023799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766154.023807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7dc00 x1862023076463104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766154.023810:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766154.023817:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076463104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766154.023828:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766154.023829:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766154.023832:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766154.023833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.023839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076463104:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766154.023846:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2655 00000100:00080000:2.0:1775766154.023853:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766154.023854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076463232:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766154.023863:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766154.023870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766154.023871:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766154.023879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7c000 x1862023076463232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.023890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076463232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766154.023894:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076463744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766154.023904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766154.023912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.023921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076463232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766154.023927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766154.023942:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766154.023951:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7e300 x1862023076463360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.023959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076463360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766154.023964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766154.023966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.023968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766154.023972:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766154.023981:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766154.023986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7f800 x1862023076463488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.023993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076463488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766154.023998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766154.023999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.024001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766154.024004:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766154.024013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766154.024024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7c700 x1862023076463616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.024050:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076463616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766154.024056:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766154.024058:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.024060:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766154.024071:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766154.024083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766154.024089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766154/real 1775766154] req@ffffa0434aa7dc00 x1862023076463744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766154.024097:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076463744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766154.024102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766154.024104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766154.024106:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076463744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766154.215099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766154.215173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766155.047091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766155.047181:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766155.239156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766155.239161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766156.071167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766156.071224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766156.263086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766156.263131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766157.095134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766157.095171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766157.287140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766157.287140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766158.119127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766158.119227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766158.311136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766158.311181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766159.143193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766159.143256:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143267:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766159.143274:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143287:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143302:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143310:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766159.143341:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766159.143365:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076463872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143397:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143400:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766159.143402:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143405:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143408:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143411:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143422:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766159.143431:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076464000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143467:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143469:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766159.143471:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143474:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143477:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143480:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143483:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766159.143491:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076464128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143513:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143518:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766159.143519:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143524:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143528:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143532:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143537:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766159.143546:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076464256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143554:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143556:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766159.143557:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143571:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143578:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766159.143586:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076463872:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766159.143591:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143603:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766159.143616:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076464384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143624:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143628:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766159.143629:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143632:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143634:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143637:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143640:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766159.143650:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076464512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143657:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143660:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766159.143661:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143664:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143676:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143679:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766159.143687:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143692:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766159.143709:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076464640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766159.143727:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143730:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766159.143732:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143736:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143740:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766159.143743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349640380 x1862023076463872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766159.143748:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143781:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766159.143805:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076464768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766159.143807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076463872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766159.143856:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766159.143860:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766159.143861:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766159.143866:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2660 00000100:00080000:0.0:1775766159.143869:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766159.143874:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766159.143887:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766159.143890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766159.143901:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076464896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766159.143907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.143927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076463872:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766159.143948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766159.144002:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144016:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349641880 x1862023076464000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144031:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076464000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144041:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144044:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144048:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766159.144054:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766159.144182:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144198:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349641500 x1862023076464128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076464128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144226:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144230:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144234:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766159.144293:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766159.144336:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349640e00 x1862023076464256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144382:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076464256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144509:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144513:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766159.144533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766159.144592:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144604:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349640700 x1862023076464384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144633:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076464384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144642:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144648:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076464384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766159.144655:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766159.144675:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349643800 x1862023076464512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076464512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144703:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766159.144726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766159.144741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349641c00 x1862023076464640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076464640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766159.144807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766159.144827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349640a80 x1862023076464768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144848:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076464768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766159.144866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766159.144907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766159.144919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766159/real 1775766159] req@ffffa04349642680 x1862023076464896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766159.144933:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076464896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766159.144942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766159.144945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766159.144956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076464896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766159.335145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766159.335176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766160.167162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766160.167231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766160.359121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766160.359122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766161.191210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766161.191260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766161.383214:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766161.383214:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766162.215166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766162.215314:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766162.407127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766162.407238:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766163.239190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766163.239195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766163.431137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766163.431344:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766164.263120:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263130:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766164.263152:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263161:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263171:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263178:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766164.263218:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076465024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775766164.263221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766164.263251:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263256:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766164.263259:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263265:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263271:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263276:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263288:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766164.263291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076465024:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766164.263308:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076465152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766164.263319:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263323:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766164.263325:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263329:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263332:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263336:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263339:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766164.263348:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076465280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766164.263367:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263370:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766164.263372:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263375:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263387:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263391:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263396:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766164.263403:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766164.263409:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076465408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766164.263418:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263421:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766164.263423:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263428:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263434:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263437:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263441:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766164.263462:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076465536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766164.263471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349643b80 x1862023076465024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766164.263479:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263504:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766164.263509:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263516:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00100000:1.0:1775766164.263521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076465024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766164.263560:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766164.263561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766164.263586:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766164.263587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.263605:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076465024:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766164.263607:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766164.263649:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076465664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766164.263659:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766164.263663:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766164.263667:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766164.263670:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263675:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263686:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263695:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766164.263695:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263714:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766164.263724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349642300 x1862023076465152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766164.263754:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076465792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766164.263774:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766164.263776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076465152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766164.263781:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766164.263784:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263789:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263794:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766164.263809:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263822:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766164.263840:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076465920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766164.263844:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.263855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766164.263874:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766164.263875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766164.263885:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766164.263891:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766164.263902:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2665 00000100:00080000:0.0:1775766164.263907:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766164.263910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766164.263915:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766164.263919:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766164.263929:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076466048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766164.263956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.263978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349642a00 x1862023076465280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.263990:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076465280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.263998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766164.264011:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766164.264089:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264101:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349641180 x1862023076465408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076465408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264120:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264123:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264137:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766164.264144:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766164.264164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264173:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349643100 x1862023076465536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264184:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076465536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264194:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264203:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076465536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766164.264231:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766164.264277:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264295:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349642680 x1862023076465664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076465664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264321:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264325:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264328:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766164.264335:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766164.264356:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349640a80 x1862023076465792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264389:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076465792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264397:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264400:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264403:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766164.264409:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766164.264427:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264435:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349641c00 x1862023076465920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264446:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076465920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264453:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264456:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264459:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076465920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766164.264464:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766164.264479:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766164.264486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766164/real 1775766164] req@ffffa04349643800 x1862023076466048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766164.264496:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076466048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766164.264511:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766164.264514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766164.264518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766164.455125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766164.455153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766165.287205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766165.479110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766165.479110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766166.311118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766166.311172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766166.503145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766166.503145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766167.335142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766167.335183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766167.527114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766167.527154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766168.359105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766168.359151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766168.551144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766168.551144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766169.383126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766169.383184:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383190:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766169.383194:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775766169.383195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766169.383202:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383212:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383217:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766169.383253:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076466176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383278:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383295:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766169.383297:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383301:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383304:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383308:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766169.383313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076466176:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766169.383318:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766169.383330:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076466304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383338:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383341:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766169.383342:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383345:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383349:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383350:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383353:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766169.383360:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076466432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383366:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383369:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766169.383370:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383373:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383375:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383378:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383387:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766169.383398:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076466560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766169.383406:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766169.383406:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383411:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766169.383414:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383418:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383421:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383424:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383427:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766169.383437:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076466688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383443:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766169.383445:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766169.383447:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766169.383452:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383457:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383460:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383463:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383467:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766169.383474:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766169.383479:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076466816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383487:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383489:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766169.383495:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466688:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766169.383498:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383519:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383523:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383527:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383532:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766169.383536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766169.383544:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076466944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383553:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383555:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766169.383557:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766169.383560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766169.383564:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383569:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383572:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766169.383576:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766169.383595:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076467072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766169.383606:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766169.383609:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766169.383611:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766169.383614:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2671 00000100:00080000:2.0:1775766169.383618:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766169.383638:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766169.383642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467072:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766169.383643:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766169.383667:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076467200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766169.383707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766169.575111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766169.575111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766170.407135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766170.407337:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766170.599136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766170.599136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766171.431187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766171.431344:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766171.623124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766172.455230:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766172.455434:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775766172.455463:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775766172.455519:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766172.455535:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766172.455583:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766172.455624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7f480 x1862023076467200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766172.455630:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766172.455661:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766172.455678:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766172.455679:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076467200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766172.455692:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:0.0:1775766172.455700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.455712:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000800:00000100:1.0:1775766172.455718:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766172.455724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766172.455738:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7f100 x1862023076466816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766172.455743:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766172.455775:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766172.455781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076466816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.455816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.455822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.455827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766172.455839:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7e680 x1862023076466560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.455863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076466560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.455873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.455877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.455882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766172.455889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7ea00 x1862023076466688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.455904:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076466688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.455914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.455918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.455922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766172.455930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7fb80 x1862023076466304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.455960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076466304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.455981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.455986:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.455990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766172.455998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7ed80 x1862023076466432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.456012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076466432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.456020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.456064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.456071:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076466432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766172.456081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7ce00 x1862023076466176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.456101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076466176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.456109:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.456114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.456118:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076466176:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766172.456158:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7d880 x1862023076467072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.456173:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076467072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.456193:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.456198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.456202:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766172.456210:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766169/real 1775766172] req@ffffa0434aa7c380 x1862023076466944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.456226:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076466944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.456235:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.456240:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766172.456245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076466944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766172.647109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766173.479097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766173.479212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766173.671099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766174.503134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766174.503140:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503147:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766174.503151:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766174.503159:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503167:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503174:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766174.503208:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076467328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775766174.503225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766174.503233:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503237:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766174.503239:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766174.503242:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503251:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503268:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766174.503278:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766174.503292:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076467456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766174.503303:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503306:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766174.503308:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766174.503315:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076467328:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766174.503316:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503323:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503330:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766174.503336:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766174.503348:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076467584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766174.503360:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503364:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766174.503366:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766174.503370:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503382:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503387:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766174.503393:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766174.503397:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766174.503411:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076467712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766174.503421:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503430:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766174.503433:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766174.503438:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503443:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503448:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766174.503456:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766174.503460:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350932300 x1862023076467328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766174.503480:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076467840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766174.503485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076467328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766174.503497:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503508:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766174.503509:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766174.503511:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766174.503520:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766174.503521:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503529:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766174.503530:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076467328:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766174.503538:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766174.503545:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467456:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766174.503546:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766174.503569:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076467968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766174.503580:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503584:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766174.503586:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766174.503592:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766174.503596:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503603:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503608:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766174.503615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350932a00 x1862023076467456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766174.503628:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766174.503634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076467456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.503645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766174.503646:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076468096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766174.503655:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.503669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467456:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766174.503670:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503678:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766174.503682:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766174.503687:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503693:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503698:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766174.503703:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766174.503711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766174.503718:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076468224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766174.503732:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766174.503736:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766174.503737:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766174.503742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766174.503746:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2676 00000100:00080000:1.0:1775766174.503752:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766174.503757:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766174.503762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350931c00 x1862023076467584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766174.503765:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766174.503782:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076467584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766174.503784:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076468352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766174.503817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.503825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.503832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766174.503840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766174.503861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.503872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350930700 x1862023076467712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.503901:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076467712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.503910:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.503914:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.503917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766174.503939:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766174.503961:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.503972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350931880 x1862023076467840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.503985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076467840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.503994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.503998:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.504001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076467840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766174.504006:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766174.504022:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.504069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350932680 x1862023076467968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.504079:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076467968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.504101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.504104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.504108:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076467968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766174.504114:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766174.504134:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.504144:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350930a80 x1862023076468096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.504156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076468096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.504164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.504168:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.504171:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766174.504194:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766174.504212:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.504228:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350930000 x1862023076468224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.504239:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076468224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.504247:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.504251:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.504261:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766174.504268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766174.504291:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766174.504304:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766174/real 1775766174] req@ffffa04350930e00 x1862023076468352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766174.504316:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076468352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766174.504324:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766174.504327:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766174.504330:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076468352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766174.695120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766175.527124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766175.527416:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766175.719117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766176.551174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766176.551221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766176.743134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766177.575189:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766177.575268:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766177.767181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766178.599213:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766178.599240:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766178.791144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766179.623121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766179.623146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766179.623324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076468480:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775766179.623421:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.623482:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825ce00 x1862023076468480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.623502:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076468480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.623513:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.623519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.623526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076468480:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766179.623559:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766179.623596:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.623610:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825d880 x1862023076468608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.623625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076468608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.623634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.623637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.623640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766179.623646:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766179.623680:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.623691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825dc00 x1862023076468736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.623734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076468736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.623761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.623767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.623771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766179.623794:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766179.623823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.623842:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825d500 x1862023076468864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.623865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076468864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.623917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.623925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.623933:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766179.623945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766179.623976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.623993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825f100 x1862023076468992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.624020:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076468992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.624100:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.624107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.624112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076468992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766179.624158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766179.624188:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.624203:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825e300 x1862023076469120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.624217:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076469120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.624242:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.624246:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.624249:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766179.624254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766179.624272:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.624282:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825ed80 x1862023076469248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.624311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076469248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.624320:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.624324:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.624337:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766179.624344:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766179.624371:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.624385:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825c000 x1862023076469376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.624426:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076469376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.624435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.624438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.624443:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766179.624449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766179.624467:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766179.624478:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766179/real 1775766179] req@ffffa0435825e680 x1862023076469504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766179.624490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076469504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766179.624498:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766179.624502:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766179.624505:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076469504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766179.815114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766180.647112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766180.647194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766180.839103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766181.671090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766181.671150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766181.863099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766182.695151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766182.695355:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766182.887109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766183.719091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766183.719194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766183.911114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766184.743111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766184.743169:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1775766184.743176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766184.743179:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766184.743185:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743197:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743209:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766184.743216:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766184.743265:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076469632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766184.743300:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743303:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766184.743305:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743308:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743311:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766184.743316:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743328:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766184.743342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076469632:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766184.743351:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076469760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766184.743367:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743372:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766184.743374:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743381:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743404:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766184.743410:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766184.743416:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743425:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766184.743448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350930e00 x1862023076469632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766184.743463:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076469888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766184.743470:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076469632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766184.743527:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766184.743528:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766184.743537:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766184.743539:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743544:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743549:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766184.743553:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766184.743556:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743562:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766184.743572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076469632:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766184.743575:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076470016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766184.743591:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743595:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766184.743596:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743600:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743604:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766184.743609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076469760:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766184.743619:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743624:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766184.743637:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076470144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766184.743639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766184.743654:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766184.743659:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350930000 x1862023076469760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766184.743664:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766184.743670:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766184.743682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076469760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766184.743689:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743707:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766184.743709:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766184.743713:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766184.743720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.743725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076469760:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766184.743725:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766184.743742:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076469888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766184.743750:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076470272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766184.743760:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743765:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766184.743769:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766184.743770:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766184.743775:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743786:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766184.743793:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743799:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766184.743802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350930a80 x1862023076469888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766184.743816:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076470400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766184.743837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076469888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766184.743845:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.743848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.743850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076469888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766184.743870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470016:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766184.743880:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743884:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766184.743887:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766184.743893:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.743900:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766184.743908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766184.743912:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.743918:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766184.743934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350932680 x1862023076470016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766184.743934:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076470528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766184.743952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076470016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766184.743957:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766184.743969:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766184.743978:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766184.743981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.743986:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.743994:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470016:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766184.743998:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2686 00000100:00080000:1.0:1775766184.744011:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766184.744017:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766184.744023:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766184.744060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766184.744083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766184.744096:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350931880 x1862023076470144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766184.744112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076470144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766184.744118:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076470656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766184.744127:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.744137:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.744140:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766184.744149:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766184.744165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766184.744172:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350930700 x1862023076470272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766184.744179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076470272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766184.744191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.744193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.744196:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766184.744200:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766184.744214:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766184.744220:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350931c00 x1862023076470400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766184.744228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076470400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766184.744233:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.744236:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.744238:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766184.744242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766184.744252:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766184.744258:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350932a00 x1862023076470528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766184.744266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076470528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766184.744271:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.744273:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.744279:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766184.744292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766184.744305:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766184.744312:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766184/real 1775766184] req@ffffa04350932300 x1862023076470656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766184.744319:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076470656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766184.744325:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766184.744327:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766184.744329:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076470656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766184.935142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766185.767182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766185.767261:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766185.959134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766186.791213:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766186.983138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766187.815121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766187.815204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766188.007154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766188.839137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766189.031121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766189.863163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766189.863164:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863182:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766189.863187:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775766189.863191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766189.863204:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863217:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863226:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766189.863273:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076470784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863299:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863302:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766189.863303:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863307:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00100000:1.0:1775766189.863328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076470784:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766189.863331:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863339:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863352:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766189.863370:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076470912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863384:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863388:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766189.863391:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863395:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863400:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766189.863402:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766189.863409:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863414:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766189.863426:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076471040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863434:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863437:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766189.863438:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863441:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863445:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863448:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863452:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766189.863452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349640700 x1862023076470784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766189.863463:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076471168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766189.863482:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076470784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766189.863484:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863490:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766189.863492:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863501:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863506:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766189.863509:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766189.863510:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863517:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766189.863518:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863524:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076470784:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766189.863529:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076471296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863536:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863539:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766189.863541:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863543:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863545:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863548:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863551:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766189.863551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766189.863565:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076471424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863573:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766189.863576:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766189.863580:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766189.863580:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766189.863584:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00080000:0.0:1775766189.863596:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863600:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766189.863605:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766189.863605:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349640e00 x1862023076470912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766189.863619:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076471552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766189.863621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076470912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766189.863643:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766189.863645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766189.863646:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766189.863651:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766189.863652:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766189.863657:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00100000:1.0:1775766189.863660:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076470912:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766189.863661:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863668:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863671:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471040:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766189.863673:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766189.863684:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076471680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766189.863691:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766189.863693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863700:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349641500 x1862023076471040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766189.863708:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766189.863712:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766189.863720:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2691 00000100:00100000:1.0:1775766189.863722:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076471040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766189.863725:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766189.863732:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766189.863733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766189.863736:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766189.863738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766189.863750:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076471808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766189.863751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766189.863771:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863798:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349641880 x1862023076471168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.863808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076471168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.863820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.863822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766189.863829:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766189.863843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349640380 x1862023076471296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.863857:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076471296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.863863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.863866:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863868:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076471296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766189.863882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766189.863892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863898:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349640700 x1862023076471424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.863905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076471424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.863911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.863913:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766189.863923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766189.863936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349643800 x1862023076471552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.863950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076471552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.863955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.863957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.863960:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766189.863965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766189.863982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.863995:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349641c00 x1862023076471680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.864006:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076471680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.864013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.864016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.864019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766189.864046:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766189.864065:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766189.864088:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766189/real 1775766189] req@ffffa04349640a80 x1862023076471808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766189.864099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076471808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766189.864107:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766189.864110:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766189.864113:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076471808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766190.055135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766190.887149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766191.079134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766191.911149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766191.911185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766192.103125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766192.935106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766193.127096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766193.959155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766194.151161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766194.983121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766194.983122:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983130:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766194.983134:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983142:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983165:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983171:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766194.983205:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076471936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766194.983225:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983228:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766194.983229:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983232:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983235:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983238:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766194.983246:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766194.983256:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076471936:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766194.983258:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076472064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766194.983271:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983273:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766194.983274:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983277:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983280:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983282:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766194.983302:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766194.983315:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076472192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766194.983317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766194.983329:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983333:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766194.983335:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983340:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983345:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983350:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983353:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349642680 x1862023076471936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766194.983358:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766194.983374:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076472320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766194.983378:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076471936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766194.983387:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983393:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766194.983395:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983403:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:1.0:1775766194.983405:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766194.983406:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983412:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766194.983424:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766194.983427:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766194.983433:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076472448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766194.983436:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076471936:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766194.983449:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983451:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766194.983458:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983462:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983464:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983468:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983472:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472064:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766194.983474:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766194.983487:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076472576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766194.983495:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766194.983500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766194.983501:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766194.983505:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983509:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983512:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983517:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983518:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349643100 x1862023076472064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766194.983520:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766194.983533:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076472704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766194.983536:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076472064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766194.983541:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983545:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766194.983548:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766194.983556:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983561:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766194.983562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766194.983564:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766194.983570:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766194.983571:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983574:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766194.983582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766194.983588:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076472832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766194.983596:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766194.983599:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766194.983600:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766194.983605:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766194.983608:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2696 00000100:00080000:0.0:1775766194.983613:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766194.983616:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766194.983620:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766194.983624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349641180 x1862023076472192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076472192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766194.983650:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076472960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766194.983668:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983676:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983680:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766194.983686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766194.983704:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983712:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349642a00 x1862023076472320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076472320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983793:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766194.983797:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766194.983813:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349642300 x1862023076472448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076472448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076472448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766194.983853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766194.983864:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349643b80 x1862023076472576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076472576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766194.983889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766194.983898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349642680 x1862023076472704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076472704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766194.983931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766194.983941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349640a80 x1862023076472832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983953:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076472832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983958:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766194.983964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766194.983973:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766194.983978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766194/real 1775766194] req@ffffa04349641c00 x1862023076472960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766194.983984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076472960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766194.983993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766194.983995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766194.983997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076472960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766195.175109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766196.007129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766196.199122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766197.031124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766197.223098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766198.055137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766198.247166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766199.079121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766199.271112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766200.103109:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766200.103109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766200.103118:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766200.103122:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103130:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103140:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103148:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766200.103197:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076473088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103219:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103222:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766200.103223:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103226:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103229:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103244:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076473088:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766200.103253:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766200.103267:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076473216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103275:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103277:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766200.103278:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103281:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103284:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103287:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766200.103290:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766200.103296:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076473344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103303:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103305:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766200.103306:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103308:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103317:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103319:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766200.103323:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766200.103324:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766200.103333:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076473472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103340:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103347:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766200.103349:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103352:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103355:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103360:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103360:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349643800 x1862023076473088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766200.103364:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766200.103377:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076473600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766200.103383:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076473088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766200.103390:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103397:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766200.103410:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766200.103413:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766200.103415:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103418:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766200.103419:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766200.103423:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076473088:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766200.103428:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766200.103437:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076473728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103442:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103445:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766200.103447:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766200.103449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766200.103452:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103457:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103459:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766200.103467:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766200.103476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766200.103479:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076473856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103488:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766200.103490:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766200.103493:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766200.103493:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349640700 x1862023076473216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766200.103496:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:0.0:1775766200.103501:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103507:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766200.103512:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766200.103520:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076473216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766200.103522:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076473984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766200.103551:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766200.103553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766200.103554:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766200.103557:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766200.103563:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2701 00000100:00080000:1.0:1775766200.103565:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766200.103566:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766200.103571:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103574:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473216:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766200.103576:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766200.103582:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076474112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766200.103598:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766200.103630:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103640:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349640380 x1862023076473344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076473344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103656:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103658:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766200.103670:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766200.103697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349641880 x1862023076473472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103712:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076473472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103723:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766200.103732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766200.103750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103760:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349641500 x1862023076473600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076473600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103788:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103790:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076473600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766200.103796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766200.103810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349640e00 x1862023076473728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076473728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103832:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766200.103848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766200.103858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349643800 x1862023076473856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076473856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103884:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103889:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766200.103892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766200.103904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349641c00 x1862023076473984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076473984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103941:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076473984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766200.103944:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076474112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766200.103955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766200.103960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766200/real 1775766200] req@ffffa04349640a80 x1862023076474112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766200.103968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076474112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766200.103972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766200.103974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766200.103991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076474112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766200.295142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766201.127214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766201.319130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766202.343109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766203.175150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766203.175162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766203.367174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766204.199122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766204.391155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766205.223104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766205.223105:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223125:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766205.223143:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775766205.223144:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766205.223151:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223161:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223167:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766205.223203:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076474240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766205.223224:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223227:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766205.223229:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223231:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223234:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223236:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223243:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766205.223253:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076474368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766205.223261:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766205.223262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076474240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766205.223267:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766205.223270:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223273:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223275:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223278:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223280:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766205.223289:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076474496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766205.223306:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223309:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766205.223311:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223313:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223317:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223322:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766205.223322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223329:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766205.223338:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076474624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766205.223344:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223346:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766205.223347:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223350:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223351:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223354:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223356:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766205.223368:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076474752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766205.223369:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349642680 x1862023076474240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766205.223388:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223394:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766205.223396:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223399:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00100000:1.0:1775766205.223405:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076474240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766205.223407:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223415:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223420:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766205.223420:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223448:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076474240:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766205.223449:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076474880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766205.223459:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223461:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766205.223462:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223465:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223467:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223470:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223473:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474368:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766205.223476:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766205.223488:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076475008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766205.223502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766205.223512:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223514:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766205.223516:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766205.223518:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349643b80 x1862023076474368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766205.223521:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223526:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223531:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223537:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766205.223545:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076474368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766205.223546:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076475136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766205.223578:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766205.223592:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766205.223597:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775766205.223598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766205.223600:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766205.223605:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2706 00000100:00080000:0.0:1775766205.223608:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766205.223612:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766205.223624:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766205.223640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766205.223655:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076475264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766205.223683:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223695:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349642300 x1862023076474496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223705:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076474496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223711:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223716:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766205.223720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766205.223732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223737:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349642a00 x1862023076474624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076474624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223750:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223753:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766205.223757:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766205.223766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223771:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349641180 x1862023076474752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076474752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223794:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076474752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766205.223798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076474880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766205.223809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349643100 x1862023076474880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076474880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223839:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223841:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076474880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766205.223855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766205.223864:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349642680 x1862023076475008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076475008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223900:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766205.223910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766205.223931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349640a80 x1862023076475136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076475136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.223965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.223967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.223969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766205.223973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766205.223986:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766205.223992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766205/real 1775766205] req@ffffa04349641c00 x1862023076475264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766205.223999:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076475264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766205.224004:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766205.224006:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766205.224013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076475264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766205.415133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766206.247181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766206.439100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766207.271164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766207.271252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766207.463163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766208.295136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766208.295187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766208.487117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766209.319169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766209.319346:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766209.511189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766210.343142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766210.343154:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343164:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766210.343169:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343194:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343217:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343225:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766210.343227:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766210.343273:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076475392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766210.343302:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343305:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766210.343307:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343310:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343313:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343337:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766210.343349:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766210.343368:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076475520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766210.343369:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076475392:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766210.343394:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343397:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766210.343398:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343404:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343409:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343411:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766210.343415:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766210.343444:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076475648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766210.343455:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343458:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766210.343460:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343463:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2712 00000800:00000100:1.0:1775766210.343472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766210.343476:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343490:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766210.343498:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766210.343511:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076475776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766210.343517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349643800 x1862023076475392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766210.343538:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343560:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766210.343563:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343571:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00100000:1.0:1775766210.343575:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076475392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766210.343633:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766210.343637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766210.343643:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766210.343645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.343654:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076475392:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766210.343658:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766210.343679:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076475904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766210.343681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766210.343706:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343710:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766210.343712:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766210.343720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766210.343727:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343733:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343738:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766210.343742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349640e00 x1862023076475520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766210.343744:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766210.343768:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076476032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766210.343771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076475520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766210.343818:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343820:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766210.343821:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343825:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343835:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343848:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766210.343852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766210.343881:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766210.343888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.343895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766210.343898:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076476160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766210.343910:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343914:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766210.343915:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343921:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00100000:1.0:1775766210.343921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766210.343930:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343934:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766210.343948:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766210.343955:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076476288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766210.343966:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766210.343972:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766210.343973:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766210.343979:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2712 00000100:00080000:0.0:1775766210.343985:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766210.343988:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766210.343992:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766210.344001:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766210.344007:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076476416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766210.344020:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349641500 x1862023076475648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344038:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076475648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344078:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344082:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766210.344095:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766210.344115:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344124:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349641880 x1862023076475776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344133:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076475776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344139:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344144:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766210.344162:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766210.344174:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344180:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349640380 x1862023076475904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076475904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344207:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344209:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344211:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076475904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766210.344216:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766210.344242:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344250:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349640700 x1862023076476032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344258:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076476032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344264:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344267:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766210.344273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766210.344284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344290:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349643800 x1862023076476160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344298:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076476160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344315:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344317:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344320:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766210.344324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766210.344339:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344345:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349641c00 x1862023076476288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344353:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076476288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344359:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344361:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344363:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766210.344367:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766210.344378:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766210.344384:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766210/real 1775766210] req@ffffa04349640a80 x1862023076476416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766210.344392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076476416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766210.344397:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766210.344399:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766210.344406:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076476416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766210.535107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766211.367149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766211.367216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766211.559111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766212.391176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766212.391288:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766212.583112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766213.415185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766213.415227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766213.607140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766214.439215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766214.439247:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766214.631134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766215.463210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766215.463210:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463220:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766215.463226:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775766215.463235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766215.463236:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463266:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463277:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766215.463356:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076476544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766215.463413:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463419:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766215.463423:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766215.463430:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463438:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463447:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766215.463476:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766215.463492:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076476672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766215.463495:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076476544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766215.463510:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463513:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766215.463515:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766215.463520:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463524:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463546:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766215.463551:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766215.463563:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076476800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766215.463572:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463575:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766215.463579:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766215.463580:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766215.463583:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463588:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463591:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766215.463595:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766215.463604:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076476928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766215.463612:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463615:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766215.463616:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766215.463620:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463623:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463626:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766215.463631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350932300 x1862023076476544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766215.463632:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766215.463643:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076477056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766215.463650:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766215.463658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076476544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766215.463659:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766215.463663:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766215.463671:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463677:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463681:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766215.463682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766215.463685:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766215.463691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766215.463696:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076477184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766215.463700:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076476544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766215.463713:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463715:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766215.463716:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766215.463738:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463742:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463746:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766215.463751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476672:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766215.463752:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766215.463774:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076477312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766215.463785:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463790:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766215.463792:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766215.463799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766215.463805:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:1.0:1775766215.463811:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766215.463815:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766215.463850:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766215.463856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350932a00 x1862023076476672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766215.463883:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076477440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766215.463911:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076476672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766215.463919:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766215.463928:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766215.463930:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766215.463931:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766215.463937:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2717 00000100:00080000:0.0:1775766215.463939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766215.463945:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766215.463947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766215.463954:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766215.463977:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476800:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766215.463980:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766215.464006:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076477568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766215.464063:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350931c00 x1862023076476800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076476800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464137:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766215.464151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766215.464172:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464198:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350930700 x1862023076476928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464207:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076476928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464212:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464214:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464216:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076476928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766215.464219:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766215.464230:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464235:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350931880 x1862023076477056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464277:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076477056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464290:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464300:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766215.464309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766215.464365:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350932680 x1862023076477184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464413:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076477184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464421:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464424:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766215.464451:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766215.464480:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350930a80 x1862023076477312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464494:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076477312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464508:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464510:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464512:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766215.464516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766215.464528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464554:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350930000 x1862023076477440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464562:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076477440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464569:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464571:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766215.464575:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766215.464588:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766215.464593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766215/real 1775766215] req@ffffa04350930e00 x1862023076477568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766215.464600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076477568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766215.464605:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766215.464607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766215.464615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076477568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766215.655086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766216.487142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766216.487199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766216.679144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766217.511115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766217.511175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766217.703174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766218.535207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766218.535297:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766218.727140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766219.559155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766219.559292:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766219.751234:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766220.583191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766220.583259:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1775766220.583260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766220.583277:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766220.583285:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.583303:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583321:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766220.583331:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766220.583400:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076477696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766220.583450:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.583476:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766220.583499:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.583511:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583543:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766220.583548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076477696:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766220.583561:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766220.583581:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766220.583616:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076477824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766220.583634:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766220.583642:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.583659:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766220.583663:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.583673:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583681:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766220.583687:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766220.583695:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766220.583713:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076477952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766220.583733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350930e00 x1862023076477696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766220.583751:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.583759:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766220.583764:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766220.583772:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076477696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766220.583783:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583797:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766220.583807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766220.583811:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766220.583820:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.583829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076477696:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775766220.583840:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766220.583863:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076478080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766220.583880:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766220.583887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477824:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766220.583891:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766220.583899:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.583906:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583914:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766220.583920:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766220.583927:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766220.583943:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076478208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766220.583954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766220.583974:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.583979:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766220.583982:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.583990:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.583995:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766220.584003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350930000 x1862023076477824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766220.584008:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766220.584082:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766220.584109:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076478336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766220.584112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076477824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766220.584140:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.584156:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766220.584167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766220.584168:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.584176:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.584182:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766220.584194:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766220.584208:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775766220.584217:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766220.584223:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766220.584244:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766220.584244:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076478464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766220.584279:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.584284:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766220.584288:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.584295:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.584310:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766220.584312:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766220.584323:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766220.584329:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766220.584342:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350930a80 x1862023076477952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766220.584351:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076478592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766220.584431:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766220.584447:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766220.584450:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766220.584456:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2722 00000100:00080000:1.0:1775766220.584463:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766220.584471:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584472:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076477952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766220.584505:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766220.584512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584530:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076477952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766220.584555:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076478720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766220.584560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766220.584609:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350932680 x1862023076478080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.584646:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076478080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.584654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584661:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766220.584668:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766220.584698:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584709:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350931880 x1862023076478208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.584720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076478208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.584729:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584732:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766220.584741:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766220.584756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350930700 x1862023076478336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.584775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076478336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.584783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766220.584794:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766220.584808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584816:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350931c00 x1862023076478464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.584843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076478464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.584851:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584854:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766220.584863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766220.584880:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350932a00 x1862023076478592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.584900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076478592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.584907:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.584911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.584915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766220.584945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766220.584974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766220.584991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766220/real 1775766220] req@ffffa04350932300 x1862023076478720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766220.585009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076478720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766220.585074:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766220.585078:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766220.585082:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076478720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766220.775194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766221.607190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766221.607276:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766221.799255:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766222.631103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766222.823172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766223.655122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766223.655156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766223.847155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766224.679130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766224.871222:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766225.703196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766225.703208:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703221:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766225.703228:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703242:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703257:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775766225.703259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766225.703268:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766225.703323:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076478848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766225.703350:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703354:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766225.703355:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703359:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703362:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703366:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703377:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766225.703388:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076478976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766225.703409:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076478848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766225.703451:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703454:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766225.703456:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703460:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703464:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703467:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703473:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766225.703486:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076479104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766225.703494:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703497:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766225.703499:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703502:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703519:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703522:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703526:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775766225.703533:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766225.703537:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076479232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766225.703545:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703547:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766225.703549:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703554:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703563:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703567:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703573:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766225.703588:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076479360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766225.703589:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350932300 x1862023076478848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766225.703621:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703636:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766225.703641:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703653:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703658:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703661:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703666:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766225.703670:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076478848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766225.703678:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076479488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766225.703709:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766225.703713:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766225.703720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766225.703722:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766225.703727:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766225.703732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076478848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766225.703738:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703748:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703754:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703759:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766225.703772:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076479616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766225.703781:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766225.703800:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766225.703804:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766225.703806:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703810:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703816:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703821:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766225.703827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703831:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766225.703843:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076479744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766225.703850:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766225.703855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350932a00 x1862023076478976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766225.703856:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766225.703859:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766225.703868:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2727 00000100:00080000:1.0:1775766225.703873:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766225.703886:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766225.703891:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766225.703905:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076479872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766225.703922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076478976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.703933:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.703937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.703940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076478976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766225.703951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766225.703974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.703984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350931c00 x1862023076479104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.703996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076479104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704003:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704006:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766225.704015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766225.704058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350930700 x1862023076479232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704080:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076479232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704109:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766225.704115:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766225.704135:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704147:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350931880 x1862023076479360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076479360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704166:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076479360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766225.704178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766225.704195:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704205:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350932680 x1862023076479488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704217:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076479488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704225:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704228:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766225.704259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766225.704279:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704289:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350930a80 x1862023076479616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704300:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076479616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704307:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704310:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704313:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766225.704319:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766225.704334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350930000 x1862023076479744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076479744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704359:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704362:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704365:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766225.704370:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766225.704383:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766225.704399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766225/real 1775766225] req@ffffa04350930e00 x1862023076479872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766225.704409:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076479872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766225.704417:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766225.704421:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766225.704424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076479872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766225.895172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766226.727120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766226.919233:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766227.751243:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766227.751392:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766227.943125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766228.775177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766228.967147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766229.799210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766229.799217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766229.991165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766230.823189:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823198:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775766230.823198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766230.823204:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823214:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823224:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823229:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766230.823264:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076480000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766230.823302:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823307:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766230.823309:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823313:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823318:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823323:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823346:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766230.823356:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076480000:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766230.823360:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076480128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766230.823371:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823374:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766230.823375:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823378:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823381:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823398:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823403:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766230.823412:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076480256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766230.823420:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823422:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766230.823424:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823427:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2732 00000800:00000100:1.0:1775766230.823432:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766230.823433:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823438:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823442:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766230.823450:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076480384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766230.823467:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823470:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766230.823476:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823481:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00100000:1.0:1775766230.823481:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349642680 x1862023076480000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766230.823488:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823495:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823500:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766230.823506:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076480000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766230.823511:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076480512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766230.823527:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766230.823529:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766230.823534:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766230.823535:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766230.823539:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766230.823542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076480000:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766230.823546:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823550:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823552:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823555:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766230.823562:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076480640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766230.823568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766230.823572:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823576:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766230.823577:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823582:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823584:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823586:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823590:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766230.823598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766230.823606:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076480768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766230.823611:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349643100 x1862023076480128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766230.823620:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823629:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766230.823632:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766230.823637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076480128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766230.823638:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823649:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823656:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823664:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766230.823665:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766230.823677:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076480896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766230.823677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766230.823689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766230.823692:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766230.823695:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766230.823696:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766230.823701:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2732 00000100:00080000:0.0:1775766230.823705:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766230.823708:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766230.823711:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766230.823721:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076481024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766230.823729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349641180 x1862023076480256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076480256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.823756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.823761:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766230.823765:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766230.823776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349642a00 x1862023076480384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823788:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076480384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.823793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.823797:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766230.823800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766230.823809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349642300 x1862023076480512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076480512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.823847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.823854:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076480512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766230.823874:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766230.823895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349643b80 x1862023076480640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076480640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.823918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.823923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766230.823927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766230.823938:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349642680 x1862023076480768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076480768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.823965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.823968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.823970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766230.823974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766230.823986:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.823992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349640a80 x1862023076480896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.823999:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076480896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.824004:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.824005:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.824007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076480896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766230.824011:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766230.824020:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766230.824047:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766230/real 1775766230] req@ffffa04349641c00 x1862023076481024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766230.824060:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076481024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766230.824068:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766230.824071:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766230.824080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766231.015139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766231.847207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766231.847305:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766232.039127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766232.871183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766233.063107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766233.895129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766233.895158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766234.087118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766234.919144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766235.111189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766235.943134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766235.943172:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943178:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766235.943183:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775766235.943206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766235.943222:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943237:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943245:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766235.943295:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076481152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943324:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943327:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766235.943329:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943332:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943350:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943355:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943369:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766235.943376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076481152:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766235.943387:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076481280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943401:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943405:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766235.943407:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943410:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943414:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943417:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943421:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766235.943430:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076481408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943448:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943450:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766235.943452:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943455:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943459:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943462:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943467:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766235.943478:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076481536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943487:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943489:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766235.943491:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943494:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00100000:1.0:1775766235.943497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766235.943498:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943505:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943508:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766235.943526:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076481664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943535:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943539:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775766235.943539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481408:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766235.943544:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943549:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943560:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766235.943563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481536:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766235.943568:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943574:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766235.943587:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076481792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766235.943594:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481664:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766235.943595:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943601:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766235.943603:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943607:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943612:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943615:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943619:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766235.943619:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766235.943632:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076481920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766235.943646:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943649:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766235.943651:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943654:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943657:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943659:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943663:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766235.943670:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076482048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766235.943680:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766235.943687:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766235.943694:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766235.943697:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766235.943701:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2737 00000100:00080000:0.0:1775766235.943705:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766235.943707:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766235.943712:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766235.943722:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076482176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766235.943733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766235.943757:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766236.135094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766236.967153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766237.159120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766237.991208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766237.991238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766238.183155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766239.015166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775766239.015445:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775766239.015494:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775766239.015522:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015585:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015603:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015614:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015624:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015634:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015647:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015660:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766239.015672:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766239.015686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349642680 x1862023076482176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076482176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.015739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.015745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766239.015763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349640a80 x1862023076482048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076482048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.015797:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.015802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766239.015811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349641c00 x1862023076481920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076481920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015832:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.015862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.015867:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766239.015877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349640e00 x1862023076481792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015897:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076481792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015907:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.015911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.015915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076481792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766239.015922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349641500 x1862023076481664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076481664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.015954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.015958:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766239.015967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349640700 x1862023076481280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.015983:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076481280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.015994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.016009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.016014:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766239.016023:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349640380 x1862023076481408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.016072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076481408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.016082:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.016087:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.016091:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766239.016100:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349641880 x1862023076481536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.016115:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076481536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.016126:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.016130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.016134:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076481536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766239.016142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766235/real 1775766239] req@ffffa04349643800 x1862023076481152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766239.016159:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076481152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766239.016169:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766239.016182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766239.016186:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076481152:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766239.207141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766240.039154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766240.039232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766240.231156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766241.063099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766241.063167:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063173:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766241.063176:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063185:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063196:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766241.063204:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766241.063243:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076482304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766241.063297:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063302:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766241.063304:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063309:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063315:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766241.063320:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063331:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766241.063340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076482304:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766241.063344:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076482432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766241.063355:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063375:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766241.063376:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063379:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063401:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766241.063406:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063412:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766241.063413:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766241.063433:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076482560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766241.063445:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063449:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766241.063451:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063455:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063460:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766241.063461:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350930e00 x1862023076482304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063466:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063475:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766241.063492:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076482688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766241.063498:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076482304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063514:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063520:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766241.063522:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766241.063523:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766241.063533:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766241.063535:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00100000:0.0:1775766241.063543:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076482304:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766241.063545:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766241.063555:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063569:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766241.063576:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766241.063585:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076482816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766241.063599:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063603:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766241.063606:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766241.063610:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766241.063621:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063628:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766241.063631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350930000 x1862023076482432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063641:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766241.063655:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076482432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766241.063656:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766241.063671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.063677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766241.063679:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076482944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766241.063682:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766241.063696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766241.063700:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063705:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766241.063707:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063724:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:1.0:1775766241.063732:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766241.063737:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766241.063741:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063749:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766241.063755:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350930a80 x1862023076482560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766241.063765:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076483072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766241.063774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076482560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063785:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766241.063789:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766241.063800:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766241.063806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766241.063810:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00080000:0.0:1775766241.063815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.063820:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766241.063822:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766241.063833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482688:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766241.063834:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766241.063845:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766241.063863:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076483200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766241.063871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766241.063884:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766241.063885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350932680 x1862023076482688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063895:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766241.063898:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766241.063907:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2742 00000100:00100000:0.0:1775766241.063910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076482688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766241.063916:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766241.063926:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766241.063927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766241.063933:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766241.063934:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.063941:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766241.063952:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076483328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766241.063975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766241.064000:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766241.064013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350931880 x1862023076482816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766241.064024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076482816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766241.064061:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.064064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.064076:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076482816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766241.064082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766241.064105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766241.064116:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350930700 x1862023076482944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766241.064128:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076482944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766241.064136:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.064139:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.064142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076482944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766241.064148:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766241.064166:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766241.064176:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350931c00 x1862023076483072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766241.064189:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076483072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766241.064197:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.064200:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.064203:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766241.064209:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766241.064228:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766241.064248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350932a00 x1862023076483200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766241.064260:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076483200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766241.064267:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.064269:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.064271:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766241.064276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766241.064313:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766241.064324:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766241/real 1775766241] req@ffffa04350932300 x1862023076483328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766241.064338:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076483328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766241.064347:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766241.064351:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766241.064354:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076483328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766241.255113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766242.087155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766242.087158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766242.279091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766243.111146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766243.303108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766244.135115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766244.135457:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766244.327114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766245.159164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766245.351089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766246.183136:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766246.183136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766246.183144:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766246.183147:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183156:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183167:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183181:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766246.183194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766246.183219:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076483456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766246.183269:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766246.183284:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766246.183287:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183293:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183298:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183304:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183310:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076483456:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766246.183315:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766246.183328:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076483584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766246.183337:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766246.183339:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766246.183341:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183344:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183347:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183349:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766246.183353:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766246.183362:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076483712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766246.183372:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766246.183377:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766246.183378:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766246.183388:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183394:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183399:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183404:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766246.183411:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766246.183421:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349643b80 x1862023076483456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766246.183424:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076483840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766246.183456:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766246.183460:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766246.183462:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766246.183466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076483456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766246.183471:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183477:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183483:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766246.183484:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.183492:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766246.183493:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766246.183498:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076483456:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766246.183510:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076483968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766246.183531:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766246.183534:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766246.183537:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766246.183539:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183543:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2747 00000800:00000100:1.0:1775766246.183553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766246.183556:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183561:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349642300 x1862023076483584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766246.183567:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766246.183578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076483584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183585:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766246.183586:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076484096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766246.183592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183599:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766246.183608:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766246.183610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766246.183614:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766246.183617:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183621:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183625:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183628:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766246.183632:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766246.183641:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766246.183642:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076484224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766246.183651:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766246.183654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349642a00 x1862023076483712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766246.183662:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766246.183665:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183673:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183688:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766246.183691:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076483712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766246.183696:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766246.183707:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766246.183709:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076484352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766246.183713:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183718:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766246.183728:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766246.183730:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483840:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766246.183734:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766246.183737:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766246.183746:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2747 00000100:00080000:0.0:1775766246.183752:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766246.183756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766246.183759:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766246.183764:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766246.183767:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349641180 x1862023076483840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766246.183777:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076484480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766246.183779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076483840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183797:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.183800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183806:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766246.183822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766246.183835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349643100 x1862023076483968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766246.183847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076483968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.183864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076483968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766246.183870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766246.183891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183898:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349643b80 x1862023076484096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766246.183905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076484096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.183913:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766246.183918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766246.183929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349643800 x1862023076484224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766246.183950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076484224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.183957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.183963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766246.183967:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766246.183980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766246.183986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349641880 x1862023076484352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766246.183994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076484352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.183999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.184000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.184002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766246.184005:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766246.184014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766246.184019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766246/real 1775766246] req@ffffa04349640380 x1862023076484480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766246.184047:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076484480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766246.184053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766246.184055:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766246.184056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076484480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766246.375137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766247.207218:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766247.207339:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766247.399153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766248.231178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766248.231271:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766248.423174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766249.255240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766249.255264:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766249.447150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766250.279190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766250.279229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766250.471190:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766251.303153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766251.303162:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303182:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766251.303208:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303237:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303252:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303258:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766251.303292:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076484608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775766251.303299:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766251.303314:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303318:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766251.303320:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303327:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303330:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303334:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303343:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766251.303350:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076484608:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766251.303356:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076484736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766251.303376:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303379:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766251.303380:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303383:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303404:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303407:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303411:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766251.303415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766251.303426:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076484864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766251.303433:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303436:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766251.303437:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303440:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303443:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303446:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303450:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766251.303452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349640700 x1862023076484608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766251.303460:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076484992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766251.303469:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303474:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775766251.303474:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076484608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766251.303478:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303485:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:1.0:1775766251.303489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766251.303496:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766251.303497:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766251.303505:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766251.303505:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076484608:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766251.303513:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766251.303523:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076485120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766251.303531:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766251.303531:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766251.303540:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766251.303542:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303546:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303550:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303553:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303557:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766251.303560:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766251.303570:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076485248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766251.303591:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349641500 x1862023076484736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766251.303600:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303613:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766251.303617:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766251.303621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076484736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766251.303629:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:1.0:1775766251.303637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766251.303655:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766251.303659:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766251.303666:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303680:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766251.303680:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766251.303696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766251.303723:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076485376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766251.303734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.303744:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349640e00 x1862023076484864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766251.303745:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303753:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766251.303756:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303765:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00100000:1.0:1775766251.303768:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076484864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766251.303771:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303779:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766251.303780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766251.303784:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766251.303785:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.303791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766251.303796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766251.303800:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076485504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766251.303807:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766251.303811:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766251.303813:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766251.303815:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2752 00000100:00080000:0.0:1775766251.303818:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766251.303820:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766251.303824:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766251.303831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766251.303835:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076485632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766251.303844:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349641c00 x1862023076484992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.303855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076484992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.303861:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.303864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.303867:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076484992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766251.303882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076485120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766251.303894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.303911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349640a80 x1862023076485120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.303921:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076485120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.303928:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.303931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.303935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076485120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766251.303940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766251.303958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.303965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349642680 x1862023076485248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.303973:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076485248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.303979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.303998:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.304002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766251.304009:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766251.304028:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.304068:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349640700 x1862023076485376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.304089:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076485376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.304108:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.304114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.304123:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766251.304131:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766251.304156:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.304167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349640380 x1862023076485504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.304177:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076485504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.304184:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.304186:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.304210:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766251.304215:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766251.304247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766251.304257:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766251/real 1775766251] req@ffffa04349641880 x1862023076485632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766251.304267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076485632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766251.304276:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766251.304280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766251.304288:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076485632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766251.495119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766252.327240:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766252.327324:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766252.519162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766253.351107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766253.351185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766253.543097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766254.375171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766254.375283:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766254.567217:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766255.399200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766255.399205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766255.591130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766256.423167:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423176:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766256.423180:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423189:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423200:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423206:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766256.423242:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076485760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423271:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423276:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766256.423279:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423286:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423290:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423294:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766256.423300:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:2.0:1775766256.423303:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766256.423324:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766256.423334:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076485888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423362:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423366:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766256.423370:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423380:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423387:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423392:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423398:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766256.423409:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076486016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423417:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423421:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766256.423422:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423430:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423434:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423439:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423442:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766256.423452:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076486144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423470:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423474:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766256.423476:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423489:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423493:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423496:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423499:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766256.423510:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076486272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423520:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423525:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766256.423528:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423533:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423536:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423542:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423549:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766256.423558:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076486400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423568:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423570:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766256.423571:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423585:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423589:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423596:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423601:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766256.423611:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076486528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423633:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423635:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766256.423637:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423641:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423646:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423651:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423656:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766256.423671:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076486656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766256.423693:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766256.423697:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766256.423700:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766256.423706:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2758 00000100:00080000:2.0:1775766256.423711:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766256.423716:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766256.423722:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766256.423733:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076486784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766256.615153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766257.447151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766257.447154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766257.639093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766258.471171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766258.663138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766259.495159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766259.495288:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766259.687107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766260.519170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766260.711128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766261.543187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766261.543281:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766261.543312:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543318:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766261.543323:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543333:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543345:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543352:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766261.543395:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076486912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543426:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543446:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766261.543449:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543454:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543459:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543465:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543476:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766261.543495:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076487040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543516:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543521:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766261.543524:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543529:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543539:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543545:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543558:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766261.543581:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076487168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543598:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543608:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766261.543613:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543624:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543630:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543639:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543661:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766261.543680:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076487296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543694:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543699:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766261.543701:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543706:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543711:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543718:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543724:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766261.543737:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076487424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543757:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543770:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766261.543772:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543781:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543789:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543798:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543807:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766261.543822:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076487552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543847:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.543865:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766261.543894:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.543903:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.543910:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.543916:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.543924:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766261.543950:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076487680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.543998:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.544012:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766261.544015:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.544033:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.544087:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.544112:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.544131:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766261.544156:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076487808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766261.544185:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766261.544190:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766261.544194:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766261.544202:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2763 00000100:00080000:1.0:1775766261.544215:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766261.544224:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766261.544233:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766261.544262:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076487936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766261.735172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766262.567175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766262.759152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766263.591126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766263.591164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766263.783149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766264.615208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766264.807215:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766265.639160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766265.639255:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766265.831118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766266.663204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766266.663220:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663229:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766266.663234:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663256:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663290:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663298:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766266.663341:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076488064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663380:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663386:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766266.663389:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663397:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663405:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663413:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663427:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766266.663454:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076488192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663479:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663485:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766266.663488:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663494:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663501:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663507:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663515:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766266.663541:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076488320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663556:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663571:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766266.663575:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663589:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663605:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663615:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663628:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766266.663654:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076488448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663711:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663717:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766266.663721:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663735:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663747:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663765:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663779:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766266.663805:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076488576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663828:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663842:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766266.663863:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663872:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663880:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663890:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663909:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766266.663929:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076488704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.663956:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.663960:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766266.663963:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.663973:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.663982:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.663988:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.663996:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766266.664016:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076488832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.664035:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.664039:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766266.664102:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.664112:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.664118:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.664122:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.664129:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766266.664159:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076488960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766266.664170:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766266.664177:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766266.664188:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766266.664200:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2768 00000100:00080000:2.0:1775766266.664208:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766266.664213:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766266.664221:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766266.664239:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076489088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766266.855100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766267.687223:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766267.687347:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766267.879169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766268.711146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766269.735169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766269.735293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766269.927143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766270.759109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766270.951128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766271.783160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766271.783167:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783176:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766271.783181:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783194:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783208:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783216:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766271.783230:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766271.783285:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076489216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783327:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783332:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766271.783335:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783341:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783347:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783353:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783367:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766271.783398:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076489344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783444:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783464:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766271.783466:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783475:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783482:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783496:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783502:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766271.783515:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076489472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783522:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783525:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766271.783528:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783531:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783534:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783543:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783548:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766271.783562:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076489600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783584:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783587:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766271.783589:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783602:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783613:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783622:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783634:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766271.783662:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076489728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783679:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783683:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766271.783685:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783690:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783698:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783703:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783709:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766271.783722:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076489856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783750:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783763:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766271.783767:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783774:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783784:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783788:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783792:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766271.783809:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076489984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783856:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783859:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766271.783861:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783867:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783871:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.783885:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.783891:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766271.783905:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076490112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766271.783937:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766271.783958:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766271.783964:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766271.783976:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2773 00000100:00080000:2.0:1775766271.783985:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766271.784009:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766271.784015:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766271.784091:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076490240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766271.975137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766272.807137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766272.999101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766273.831146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766273.831226:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766274.023191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766274.855153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766275.047108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766275.879126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766275.879203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766276.071111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766276.903105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766276.903107:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903116:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766276.903119:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903130:0:52459:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903146:0:52459:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903152:0:52459:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766276.903181:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076490368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903202:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903205:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766276.903206:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903209:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903212:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903214:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903222:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766276.903234:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076490496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903247:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903250:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766276.903251:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903254:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903256:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903258:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903261:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766276.903270:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076490624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903275:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903282:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766276.903284:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903286:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903290:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903295:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903302:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766276.903312:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076490752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903318:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903321:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766276.903322:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903325:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903328:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903331:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903336:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766276.903347:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076490880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903356:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903371:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766276.903374:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903381:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903387:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903390:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903400:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766276.903411:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076491008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903421:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903426:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766276.903429:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903433:0:52459:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903437:0:52459:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903444:0:52459:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903451:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766276.903461:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076491136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903475:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903482:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766276.903485:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903492:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903496:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903502:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903511:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766276.903520:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076491264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766276.903530:0:52459:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766276.903533:0:52459:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766276.903541:0:52459:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766276.903550:0:52459:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2778 00000100:00080000:2.0:1775766276.903553:0:52459:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766276.903573:0:52459:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766276.903582:0:52459:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766276.903598:0:52459:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076491392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766277.095089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766277.927099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766277.927115:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766278.119132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766278.951126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766279.143093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766279.975147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766279.975197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766280.167142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766280.999167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766281.191164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766282.023164:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023179:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766282.023186:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023201:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023216:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023227:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766282.023289:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076491520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775766282.023290:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766282.023338:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023344:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766282.023346:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023352:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023356:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023362:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023376:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766282.023398:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076491648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766282.023403:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076491520:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766282.023448:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023452:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766282.023454:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023460:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023465:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023470:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023477:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766282.023491:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076491776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766282.023511:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023516:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766282.023518:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023527:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2783 00000800:00000100:2.0:1775766282.023527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766282.023531:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023537:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023542:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766282.023555:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076491904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766282.023565:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023569:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766282.023572:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023576:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023605:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023612:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023620:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766282.023627:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350930e00 x1862023076491520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766282.023644:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076492032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766282.023664:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023670:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775766282.023675:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076491520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766282.023676:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023716:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023725:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023736:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023748:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775766282.023751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766282.023762:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076492160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766282.023766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.023782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076491520:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766282.023784:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023796:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766282.023812:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023819:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.023822:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023826:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023833:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766282.023836:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766282.023849:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076492288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766282.023880:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766282.023915:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.023920:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766282.023922:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766282.023929:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00100000:2.0:1775766282.023935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350930000 x1862023076491648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766282.023939:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.023951:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766282.023964:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766282.023970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076491648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766282.023983:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076492416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766282.024009:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766282.024018:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766282.024021:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766282.024023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766282.024057:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2783 00000100:00080000:1.0:1775766282.024062:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766282.024069:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766282.024074:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775766282.024095:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766282.024102:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766282.024123:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766282.024124:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076492544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766282.024177:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350930a80 x1862023076491776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024210:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076491776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024221:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024226:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766282.024237:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766282.024257:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024268:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350932680 x1862023076491904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024282:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076491904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024291:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024310:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076491904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766282.024318:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766282.024343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024359:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350931880 x1862023076492032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024383:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076492032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024399:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024405:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024412:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766282.024424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766282.024463:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024483:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350930700 x1862023076492160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076492160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024510:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766282.024545:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766282.024569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350931c00 x1862023076492288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076492288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766282.024633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766282.024654:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024664:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350932a00 x1862023076492416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024679:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076492416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024700:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024706:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766282.024716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766282.024752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766282.024773:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766282/real 1775766282] req@ffffa04350932300 x1862023076492544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766282.024797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076492544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766282.024819:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766282.024823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766282.024827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076492544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766282.215115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766283.239200:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766284.071122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766284.071140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766284.263088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766285.095114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766285.287160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766286.119127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766286.119220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766286.311125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766287.143180:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766287.143186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766287.143196:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766287.143203:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143215:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00080000:1.0:1775766287.143229:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143237:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766287.143259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766287.143285:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076492672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766287.143317:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143321:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766287.143323:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143331:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00080000:1.0:1775766287.143336:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143344:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143357:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766287.143358:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076492672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766287.143385:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076492800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766287.143399:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143403:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766287.143404:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143407:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00080000:1.0:1775766287.143410:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766287.143430:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766287.143431:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143438:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766287.143453:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076492928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766287.143465:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143469:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766287.143473:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766287.143474:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350932300 x1862023076492672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766287.143497:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00100000:2.0:1775766287.143499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076492672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766287.143508:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766287.143512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766287.143522:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766287.143522:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.143534:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076492672:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775766287.143538:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766287.143552:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076493056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766287.143557:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766287.143588:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143591:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766287.143594:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143602:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2788 00000800:00000100:2.0:1775766287.143613:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766287.143615:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143622:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143628:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766287.143630:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350932a00 x1862023076492800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766287.143655:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076493184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766287.143657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076492800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766287.143696:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766287.143702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766287.143709:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766287.143712:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766287.143714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766287.143722:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00100000:2.0:1775766287.143723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766287.143730:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143735:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143742:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766287.143751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766287.143757:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076493312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766287.143767:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143771:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766287.143773:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143777:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2788 00000800:00000100:2.0:1775766287.143787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766287.143792:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143798:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143807:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766287.143811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350931c00 x1862023076492928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766287.143823:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076493440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766287.143832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076492928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.143843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766287.143861:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766287.143871:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766287.143872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766287.143877:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766287.143884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076492928:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766287.143889:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00080000:1.0:1775766287.143897:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143903:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.143916:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766287.143916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076493056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766287.143940:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076493568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766287.143949:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766287.143949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766287.143959:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766287.143962:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766287.143968:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2788 00000100:00100000:2.0:1775766287.143972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350930700 x1862023076493056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766287.143974:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766287.143989:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766287.144004:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766287.144009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076493056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766287.144018:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076493696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766287.144064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144078:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076493056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766287.144100:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076493184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766287.144120:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766287.144129:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350931880 x1862023076493184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766287.144144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076493184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.144149:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144152:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144154:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076493184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766287.144158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766287.144172:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766287.144178:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350932680 x1862023076493312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766287.144186:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076493312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.144191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766287.144198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766287.144208:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766287.144214:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350930a80 x1862023076493440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766287.144220:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076493440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.144234:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144236:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144238:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766287.144251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766287.144264:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766287.144270:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350930000 x1862023076493568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766287.144278:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076493568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.144283:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144285:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144287:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766287.144291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766287.144300:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766287.144306:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766287/real 1775766287] req@ffffa04350930e00 x1862023076493696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766287.144313:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076493696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766287.144317:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766287.144319:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766287.144326:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076493696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766287.335155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766288.167116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766288.167168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766288.359100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766289.191190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766289.191204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766289.383111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766290.215126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766290.215201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766290.407079:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766291.239128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766291.239153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766291.431140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766292.263116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766292.263138:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263146:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766292.263151:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263164:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263175:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263181:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766292.263219:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076493824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263256:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263260:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766292.263262:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263267:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263271:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263277:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263287:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766292.263303:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076493952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263316:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263320:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766292.263323:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263327:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263347:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263356:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263362:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766292.263378:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076494080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263389:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263394:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766292.263398:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263408:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263416:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263421:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263428:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766292.263447:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076494208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263459:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263466:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766292.263468:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263473:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263477:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263483:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263492:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766292.263508:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076494336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263560:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263569:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766292.263573:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263583:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263593:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263601:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263608:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766292.263626:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076494464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263664:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263673:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766292.263677:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263688:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263697:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263705:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263712:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766292.263730:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076494592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263776:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263786:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766292.263789:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263795:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263839:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263849:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263858:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766292.263879:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076494720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766292.263899:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766292.263906:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766292.263909:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766292.263916:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2793 00000100:00080000:2.0:1775766292.263925:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766292.263934:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766292.263944:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766292.263959:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076494848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766292.455127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766293.287131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766293.287189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766293.479099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766294.311101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766294.503105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766295.335129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766295.335190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766295.527137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766296.359205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766296.359378:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766296.551184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766297.383223:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766297.383262:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383273:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766297.383279:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383289:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383300:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383308:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766297.383351:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076494976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775766297.383370:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766297.383406:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383411:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766297.383413:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383418:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383424:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383441:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383456:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766297.383475:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076495104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383488:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383493:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766297.383495:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383499:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383504:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383511:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383518:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766297.383534:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076495232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383556:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383562:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766297.383565:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383575:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383589:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383599:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383608:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766297.383629:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076495360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383641:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383646:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766297.383651:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383658:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383663:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383668:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383676:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766297.383694:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076495488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383730:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383737:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766297.383740:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383787:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383796:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383805:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383816:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766297.383837:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076495616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383879:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383888:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766297.383893:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383902:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383913:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.383923:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.383931:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766297.383950:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076495744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.383967:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.383973:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766297.383977:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.383987:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.383993:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.384001:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.384010:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766297.384056:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076495872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766297.384068:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766297.384075:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766297.384078:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766297.384085:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2799 00000100:00080000:2.0:1775766297.384099:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766297.384107:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766297.384116:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766297.384135:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076496000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766297.575171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766298.407240:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766298.407284:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766298.599121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766299.431141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766299.431193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766299.623148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766300.455236:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766300.455290:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766300.647148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766301.479143:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766301.479201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766301.671169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766302.503171:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503185:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766302.503192:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503207:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00100000:1.0:1775766302.503218:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766302.503223:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503235:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766302.503242:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766302.503306:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076496128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503352:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766302.503353:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503357:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503360:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503364:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503375:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766302.503387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076496256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766302.503401:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503404:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503405:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503408:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503411:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766302.503418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076496384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503450:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766302.503453:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503467:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503475:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503480:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503490:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766302.503506:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076496512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503534:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503538:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766302.503539:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503542:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503545:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503548:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766302.503565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076496640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503572:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503575:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766302.503577:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503585:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503592:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503597:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503602:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766302.503615:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076496768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503657:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503665:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766302.503672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503681:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503689:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503696:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503727:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766302.503755:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076496896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503762:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503764:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766302.503766:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503768:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503876:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503879:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503883:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766302.503893:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076497024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766302.503908:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766302.503911:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766302.503913:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766302.503916:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2804 00000100:00080000:0.0:1775766302.503928:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766302.503934:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766302.503939:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766302.503952:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076497152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766302.695152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766303.527159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766303.527182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766303.719095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766304.551216:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766304.551330:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766304.743135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766305.575184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766305.575293:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775766305.575304:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775766305.575349:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775766305.575377:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575427:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575448:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575456:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575465:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575472:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575480:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575487:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766305.575493:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766305.767126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766306.599226:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766306.599273:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766306.791123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766307.623115:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623124:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766307.623129:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623140:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623151:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766307.623158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766307.623159:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766307.623167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766307.623196:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076497280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623250:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623254:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766307.623256:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623261:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623266:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623272:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623282:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766307.623296:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076497408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766307.623319:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623323:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623327:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623332:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623337:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766307.623351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076497536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623361:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623366:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766307.623367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623371:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623384:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623389:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766307.623413:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076497664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623427:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766307.623429:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623434:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623439:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623446:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623455:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766307.623471:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076497792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623481:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623490:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766307.623494:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623502:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623507:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623515:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766307.623541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076497920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623572:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623575:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766307.623577:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623584:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623590:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623599:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623606:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766307.623623:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076498048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623636:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623645:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766307.623649:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623687:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623695:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623704:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623713:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766307.623730:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076498176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766307.623746:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766307.623749:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766307.623751:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766307.623758:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2809 00000100:00080000:0.0:1775766307.623771:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766307.623777:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766307.623785:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766307.623797:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076498304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766307.815111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766308.647281:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766308.647357:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766308.839086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766309.671177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766309.863090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766310.695093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766310.695176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766310.887094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766311.719093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766311.911177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766312.743123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766312.743146:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743152:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766312.743156:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743163:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743172:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775766312.743178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766312.743178:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766312.743210:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076498432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743233:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743237:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766312.743239:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743244:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743248:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743254:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743264:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766312.743290:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076498560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743302:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743304:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766312.743306:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743309:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743313:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743316:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743320:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766312.743332:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076498688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743339:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743342:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766312.743343:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743346:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743348:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743350:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743353:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766312.743360:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076498816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743367:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743370:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766312.743374:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743386:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743391:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743398:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743406:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766312.743417:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076498944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743433:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743435:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766312.743436:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743439:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743442:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743447:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743450:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766312.743458:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076499072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743466:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743468:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766312.743469:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743472:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743474:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743482:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743485:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766312.743495:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076499200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743513:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743516:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766312.743520:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743529:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743537:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743544:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743552:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766312.743570:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076499328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766312.743581:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766312.743586:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766312.743590:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766312.743595:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2814 00000100:00080000:2.0:1775766312.743599:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766312.743624:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766312.743629:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766312.743647:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076499456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766312.935138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766313.767207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766313.959176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766314.791219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766314.791285:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766314.983221:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766315.815197:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766316.007170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766316.839270:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766316.839271:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766317.031177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766317.863129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766317.863180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766317.863194:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863201:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766317.863206:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863217:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863237:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863245:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766317.863294:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076499584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863321:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863324:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766317.863326:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863330:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863333:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863337:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863348:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766317.863385:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076499712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863394:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863397:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766317.863399:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863402:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863406:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863409:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863413:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766317.863420:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076499840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863427:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863430:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766317.863431:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863433:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863436:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863439:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863446:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766317.863455:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076499968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863463:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863466:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766317.863467:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863476:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863480:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863483:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863487:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766317.863499:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076500096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863511:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863518:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766317.863520:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863526:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863529:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863532:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863536:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766317.863550:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076500224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863581:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863588:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766317.863596:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863602:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863605:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863609:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863613:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766317.863626:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076500352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863645:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863653:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766317.863656:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863663:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863673:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863681:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863693:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766317.863712:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076500480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766317.863723:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766317.863729:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766317.863731:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766317.863734:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2819 00000100:00080000:0.0:1775766317.863737:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766317.863740:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766317.863744:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766317.863755:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076500608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766318.055183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766318.887208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766318.887341:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766319.079129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766319.911115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766319.911180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766320.103125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766320.935154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766321.127215:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766321.959130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766321.959282:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766322.151136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766322.983162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766322.983177:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.983188:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766322.983210:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983225:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983243:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983255:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766322.983346:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076500736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983391:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.983396:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766322.983399:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983405:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983413:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983422:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766322.983438:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766322.983468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076500736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766322.983480:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076500864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983499:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.983525:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766322.983528:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983550:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983558:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983575:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766322.983584:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766322.983604:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076500992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983634:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766322.983639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766322.983641:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766322.983647:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983654:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983661:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983667:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766322.983676:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766322.983693:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076501120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983704:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.983709:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766322.983711:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983719:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983723:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983730:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766322.983731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349641500 x1862023076500736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766322.983735:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766322.983756:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076501248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983768:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766322.983770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076500736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766322.983776:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766322.983783:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983797:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:1.0:1775766322.983802:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.983814:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766322.983815:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983824:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766322.983835:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076500736:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766322.983842:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766322.983865:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076501376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766322.983876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076500864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766322.983881:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.983887:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766322.983889:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983897:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983902:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983907:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766322.983913:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766322.983921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766322.983932:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076501504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766322.983944:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766322.983946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349640e00 x1862023076500864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766322.983955:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766322.983960:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.983974:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2824 00000100:00080000:0.0:1775766322.983983:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.983993:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766322.983993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076500864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766322.984005:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766322.984017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766322.984059:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076501632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766322.984061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984076:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076500864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766322.984092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076500992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766322.984116:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766322.984121:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766322.984124:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766322.984132:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2824 00000800:00000100:1.0:1775766322.984140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766322.984145:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766322.984155:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349641c00 x1862023076500992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766322.984167:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766322.984185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076500992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766322.984194:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076501760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766322.984206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984220:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984228:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076500992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766322.984235:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076501120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766322.984257:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349640a80 x1862023076501120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984294:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076501120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984303:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984310:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076501120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766322.984316:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076501248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766322.984334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349642680 x1862023076501248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984355:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076501248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984393:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984397:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984400:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076501248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766322.984426:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766322.984449:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984461:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349643800 x1862023076501376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984475:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076501376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984493:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984497:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984501:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766322.984565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766322.984602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349642a00 x1862023076501504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076501504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766322.984699:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766322.984726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349641180 x1862023076501632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076501632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766322.984790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766322.984824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766322.984836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766322/real 1775766322] req@ffffa04349643100 x1862023076501760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766322.984868:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076501760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766322.984884:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766322.984887:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766322.984890:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076501760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766323.175159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766324.007268:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766324.007371:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766324.199198:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766325.031246:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766325.223124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766326.055107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766326.055257:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766326.247157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766327.079143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766327.271105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766328.103196:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103208:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766328.103214:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766328.103216:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766328.103228:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103241:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103248:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766328.103259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766328.103295:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076501888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103321:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103325:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766328.103326:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766328.103330:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103333:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103336:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766328.103347:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766328.103353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076501888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766328.103361:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076502016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103369:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103371:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766328.103372:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766328.103375:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103377:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103379:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766328.103382:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766328.103423:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076502144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103430:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103433:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766328.103434:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766328.103438:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766328.103442:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103445:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103448:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766328.103452:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766328.103487:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076502272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103495:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103497:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766328.103499:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766328.103502:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103506:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766328.103509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7d500 x1862023076501888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103511:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766328.103516:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766328.103529:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076502400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103535:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766328.103539:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076501888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103539:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766328.103540:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766328.103550:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766328.103552:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103555:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103560:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766328.103564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766328.103565:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766328.103569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076501888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766328.103578:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076502528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766328.103586:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103588:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766328.103589:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766328.103591:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103594:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103596:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103599:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502016:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766328.103602:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766328.103624:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076502656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766328.103626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7df80 x1862023076502016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103639:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766328.103648:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766328.103652:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766328.103660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076502016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103660:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00080000:2.0:1775766328.103668:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766328.103671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766328.103675:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766328.103678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.103683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502016:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766328.103683:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766328.103692:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766328.103701:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076502784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766328.103711:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766328.103720:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766328.103722:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7c380 x1862023076502144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103724:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766328.103728:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766328.103738:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2829 00000100:00100000:1.0:1775766328.103741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076502144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766328.103743:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766328.103750:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766328.103752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766328.103756:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766328.103757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.103762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766328.103767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766328.103787:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076502912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766328.103810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103824:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7fb80 x1862023076502272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.103837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076502272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.103857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.103859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.103862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766328.103867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766328.103881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7ed80 x1862023076502400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.103895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076502400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.103901:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.103903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.103905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076502400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766328.103909:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766328.103918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7ce00 x1862023076502528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.103942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076502528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.103948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.103950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.103951:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766328.103966:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766328.103978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.103983:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7d880 x1862023076502656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.103991:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076502656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.103996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.103999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.104001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766328.104004:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766328.104013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.104017:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7d500 x1862023076502784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.104024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076502784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.104059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.104061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.104063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766328.104067:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766328.104082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766328.104089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766328/real 1775766328] req@ffffa0434aa7ea00 x1862023076502912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766328.104096:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076502912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766328.104102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766328.104104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766328.104106:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076502912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766328.295190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766329.127152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766329.319142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766330.151147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766330.151456:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766330.343211:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766331.175158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766331.367198:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766332.199131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766332.199166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766332.391104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766333.223162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766333.223171:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.223183:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766333.223190:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766333.223206:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.223220:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223229:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766333.223280:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076503040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766333.223317:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.223321:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766333.223323:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766333.223328:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.223332:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223339:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766333.223370:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076503040:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766333.223389:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766333.223433:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076503168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766333.223448:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.223461:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766333.223464:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766333.223473:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766333.223481:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.223489:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223495:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766333.223504:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766333.223516:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7e680 x1862023076503040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766333.223525:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076503296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766333.223544:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.223562:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766333.223570:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766333.223583:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00100000:1.0:1775766333.223589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076503040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766333.223591:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223606:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766333.223610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.223621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766333.223623:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766333.223627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076503040:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766333.223647:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076503424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766333.223658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503168:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766333.223675:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.223680:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766333.223695:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766333.223698:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766333.223711:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00100000:1.0:1775766333.223718:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7f480 x1862023076503168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766333.223719:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223735:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766333.223743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076503168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766333.223747:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766333.223764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.223769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.223772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766333.223782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766333.223782:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076503552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766333.223808:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766333.223813:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766333.223819:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766333.223823:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766333.223828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7dc00 x1862023076503296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766333.223839:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.223851:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766333.223852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076503296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766333.223867:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766333.223875:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766333.223897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.223902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.223913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766333.223913:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076503680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766333.223940:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766333.223946:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503424:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766333.223948:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766333.223957:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766333.223967:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.223974:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.223981:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766333.223983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766333.223995:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766333.224000:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7c700 x1862023076503424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766333.224024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076503424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766333.224080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766333.224087:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076503808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766333.224094:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.224106:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503424:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766333.224118:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766333.224125:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503552:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766333.224126:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766333.224132:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766333.224145:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.224154:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766333.224161:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766333.224166:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766333.224169:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766333.224181:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7f800 x1862023076503552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766333.224194:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076503936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766333.224206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076503552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766333.224214:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766333.224229:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766333.224234:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766333.224246:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766333.224248:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2834 00000100:00080000:2.0:1775766333.224259:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766333.224260:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766333.224267:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766333.224282:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076503552:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766333.224284:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766333.224309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766333.224323:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076504064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766333.224341:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766333.224358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7e300 x1862023076503680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766333.224371:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076503680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766333.224379:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.224383:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.224386:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766333.224403:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766333.224444:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766333.224471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7f100 x1862023076503808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766333.224482:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076503808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766333.224489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.224492:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.224494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766333.224500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766333.224514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766333.224530:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7c000 x1862023076503936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766333.224540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076503936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766333.224547:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.224550:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.224552:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076503936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766333.224557:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766333.224570:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766333.224577:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766333/real 1775766333] req@ffffa0434aa7dc00 x1862023076504064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766333.224586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076504064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766333.224593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766333.224595:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766333.224598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766333.415211:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766334.247156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766334.439107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766335.271138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766335.463148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766336.295116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766336.487138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766337.319125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766337.511091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766338.343137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766338.343145:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343153:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766338.343158:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343185:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343195:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343202:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766338.343243:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076504192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343267:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343270:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766338.343272:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343275:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343286:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343289:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343297:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766338.343307:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076504320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343317:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343320:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766338.343322:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766338.343331:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076504192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766338.343333:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343340:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343350:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343353:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766338.343387:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076504448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343413:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343415:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766338.343417:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343420:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343424:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343428:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343434:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766338.343442:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076504576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343448:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343450:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766338.343452:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343454:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343456:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343458:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343462:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775766338.343464:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766338.343472:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076504704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343478:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343486:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766338.343487:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343490:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343500:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343504:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343509:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766338.343523:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076504832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766338.343532:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343534:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766338.343536:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766338.343541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7f480 x1862023076504192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766338.343541:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:2.0:1775766338.343551:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766338.343562:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766338.343575:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766338.343584:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076504960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766338.343591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076504192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766338.343596:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343611:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766338.343617:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343625:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00080000:1.0:1775766338.343626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766338.343628:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766338.343632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766338.343633:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766338.343644:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076504192:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766338.343644:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766338.343668:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076505088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766338.343681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766338.343706:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766338.343710:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1775766338.343713:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766338.343716:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766338.343724:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2840 00000100:00100000:1.0:1775766338.343730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7e680 x1862023076504320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766338.343733:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766338.343762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076504320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766338.343771:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766338.343776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.343783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766338.343783:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766338.343792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766338.343802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766338.343813:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076505216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766338.343836:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.343847:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7ea00 x1862023076504448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.343859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076504448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.343880:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.343883:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.343886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766338.343893:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766338.343913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.343922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7d500 x1862023076504576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.343932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076504576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.343939:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.343942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.343945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766338.343950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766338.343962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.343969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7d880 x1862023076504704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.343978:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076504704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.343986:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.343988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.343996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076504704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766338.344017:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766338.344036:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.344075:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7ce00 x1862023076504832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.344090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076504832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.344099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.344102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.344104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766338.344109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766338.344126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.344134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7ed80 x1862023076504960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.344142:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076504960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.344148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.344150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.344152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076504960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766338.344169:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766338.344180:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.344196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7fb80 x1862023076505088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.344204:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076505088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.344210:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.344213:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.344215:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766338.344218:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766338.344230:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766338.344236:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766338/real 1775766338] req@ffffa0434aa7c380 x1862023076505216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766338.344243:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076505216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766338.344249:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766338.344251:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766338.344252:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766338.535101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766339.367151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766339.559185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766340.391154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766340.583132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766341.415133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766341.607149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766342.439116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766342.631120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766343.463126:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463135:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766343.463140:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463149:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463163:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463171:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766343.463180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766343.463214:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076505344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463238:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463242:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766343.463243:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463246:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463249:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463251:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463273:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766343.463281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076505344:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766343.463286:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076505472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463296:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463298:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766343.463299:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463302:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463305:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463323:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463327:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766343.463337:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076505600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463343:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463346:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766343.463347:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463350:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463353:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463355:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463359:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766343.463364:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766343.463369:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076505728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463377:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463380:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766343.463387:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463391:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463394:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463396:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463400:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766343.463409:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076505856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463416:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463420:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775766343.463421:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7df80 x1862023076505344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766343.463422:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463429:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463434:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463437:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463441:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766343.463446:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076505344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766343.463450:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076505984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766343.463467:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766343.463468:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463474:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766343.463475:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766343.463477:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766343.463481:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00100000:1.0:1775766343.463484:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076505344:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766343.463486:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463492:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463495:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766343.463508:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076506112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766343.463510:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766343.463523:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463525:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766343.463526:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463530:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463534:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766343.463536:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766343.463541:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463542:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766343.463552:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076506240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766343.463559:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766343.463561:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766343.463562:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766343.463564:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2845 00000100:00080000:2.0:1775766343.463566:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766343.463570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7c380 x1862023076505472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766343.463570:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766343.463575:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766343.463584:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076506368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766343.463597:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076505472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463616:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766343.463624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766343.463641:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7fb80 x1862023076505600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076505600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463672:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463675:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766343.463681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766343.463694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463701:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7ed80 x1862023076505728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076505728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463715:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766343.463720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766343.463729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463734:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7ce00 x1862023076505856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463746:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076505856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463776:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076505856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766343.463783:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766343.463805:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463813:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7d880 x1862023076505984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076505984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463827:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463829:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463831:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076505984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766343.463845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766343.463856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7d500 x1862023076506112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463869:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076506112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766343.463881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766343.463890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7ea00 x1862023076506240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076506240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766343.463922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766343.463937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766343.463946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766343/real 1775766343] req@ffffa0434aa7e680 x1862023076506368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766343.463959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076506368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766343.463968:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766343.463971:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766343.463974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076506368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766343.655134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766344.487126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766344.679124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766345.511131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766345.703133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766346.535133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766346.727099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766347.559123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766347.751144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766348.583126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766348.583295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076506496:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775766348.583347:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583389:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825ea00 x1862023076506496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583410:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076506496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076506496:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766348.583463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766348.583486:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583498:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825e680 x1862023076506624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583517:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076506624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583530:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583536:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583541:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766348.583547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766348.583564:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825c000 x1862023076506752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076506752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583625:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583629:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766348.583638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766348.583665:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825ed80 x1862023076506880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583696:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076506880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583728:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076506880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766348.583753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766348.583785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825e300 x1862023076507008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076507008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583848:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766348.583855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766348.583885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583895:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825f100 x1862023076507136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076507136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766348.583917:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766348.583929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.583935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825d500 x1862023076507264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.583955:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076507264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.583963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.583966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.583969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766348.583975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766348.583994:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.584013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825dc00 x1862023076507392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.584045:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076507392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.584055:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.584059:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.584063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766348.584080:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766348.584101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766348.584112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766348/real 1775766348] req@ffffa0435825d880 x1862023076507520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766348.584124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076507520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766348.584132:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766348.584136:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766348.584139:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076507520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766348.775115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766349.607092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766349.799134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766350.631110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766350.823151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766351.655183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766351.655191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766351.847144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766352.679103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766352.871115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766353.703106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766353.703119:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766353.703125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766353.703129:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703140:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703148:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766353.703186:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076507648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766353.703229:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703233:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703236:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703240:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766353.703248:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766353.703259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076507648:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766353.703262:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076507776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766353.703284:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703289:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703294:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703299:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766353.703305:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766353.703319:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076507904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703340:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775766353.703342:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766353.703343:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703350:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703355:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703359:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766353.703364:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766353.703374:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076508032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766353.703384:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349640700 x1862023076507648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766353.703386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766353.703399:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703408:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703414:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703419:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703423:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076507648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766353.703426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766353.703436:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703441:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766353.703447:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076508160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766353.703449:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076507648:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766353.703464:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766353.703468:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703472:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703475:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703479:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507776:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766353.703484:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766353.703496:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076508288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766353.703509:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703512:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2855 00000800:00000100:1.0:1775766353.703517:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766353.703519:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703523:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766353.703527:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766353.703533:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349642300 x1862023076507776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766353.703537:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076508416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766353.703558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076507776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766353.703560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766353.703563:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766353.703575:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766353.703585:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00100000:1.0:1775766353.703586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507776:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766353.703592:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766353.703596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507904:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766353.703599:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766353.703614:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766353.703614:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703622:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349641880 x1862023076507904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766353.703628:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076508544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766353.703637:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766353.703643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766353.703647:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766353.703653:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2855 00000100:00080000:0.0:1775766353.703658:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766353.703663:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076507904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766353.703670:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766353.703682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076507904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766353.703696:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076508672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766353.703701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766353.703736:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703747:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349640380 x1862023076508032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076508032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766353.703788:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766353.703803:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703809:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349643b80 x1862023076508160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076508160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766353.703859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766353.703873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349640700 x1862023076508288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076508288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766353.703898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766353.703907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349643100 x1862023076508416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076508416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766353.703936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766353.703948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703954:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349641180 x1862023076508544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076508544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.703968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.703969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766353.703973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766353.703982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766353.703986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766353/real 1775766353] req@ffffa04349642a00 x1862023076508672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766353.703993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076508672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766353.703998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766353.704000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766353.704002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076508672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766353.895123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766354.727081:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766354.919146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766355.751093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766355.943148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766356.775121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766356.967143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766357.799147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766357.799205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766357.991107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766358.823109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766358.823120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766358.823130:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823138:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823147:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766358.823154:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766358.823205:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076508800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823256:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766358.823258:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823262:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823268:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766358.823273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076508800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766358.823274:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766358.823303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076508928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766358.823318:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823321:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766358.823333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823337:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775766358.823338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766358.823350:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076509056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823360:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766358.823362:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823365:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823368:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766358.823371:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823380:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349643800 x1862023076508800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766358.823381:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766358.823396:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076509184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766358.823403:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076508800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766358.823406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766358.823419:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766358.823421:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766358.823425:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:2.0:1775766358.823426:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823432:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076508800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766358.823433:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766358.823443:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766358.823461:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766358.823466:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076509312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1775766358.823486:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766358.823491:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823497:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823502:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766358.823505:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349642680 x1862023076508928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823515:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076508928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823523:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823527:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823530:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076508928:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766358.823540:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766358.823548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766358.823565:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076509440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766358.823573:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766358.823578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775766358.823585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349640a80 x1862023076509056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766358.823586:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823595:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823602:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766358.823605:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076509056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766358.823609:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823617:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775766358.823618:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823624:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823628:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766358.823630:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076509568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766358.823638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509184:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766358.823641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766358.823644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766358.823645:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823649:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823652:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766358.823654:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766358.823658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766358.823660:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766358.823668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349641c00 x1862023076509184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766358.823672:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076509696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766358.823691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766358.823695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076509184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766358.823696:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766358.823700:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766358.823708:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2860 00000100:00080000:0.0:1775766358.823712:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766358.823712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766358.823718:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766358.823720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766358.823723:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766358.823724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766358.823741:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076509824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766358.823746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766358.823767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349640e00 x1862023076509312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076509312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823808:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076509312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766358.823811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766358.823824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823831:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349641500 x1862023076509440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076509440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823846:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766358.823850:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766358.823859:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349643800 x1862023076509568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076509568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823884:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766358.823892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766358.823904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349642a00 x1862023076509696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076509696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766358.823936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766358.823947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766358.823952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766358/real 1775766358] req@ffffa04349641180 x1862023076509824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766358.823959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076509824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766358.823964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766358.823965:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766358.823972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076509824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766359.015132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766359.847304:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766359.847327:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766360.039119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766360.871157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766361.063117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766361.895127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766361.895222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766362.087087:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766362.919163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766363.111170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766363.943185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766363.943195:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943205:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766363.943206:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766363.943210:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943223:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943234:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943242:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766363.943289:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076509952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766363.943316:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943319:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766363.943321:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943324:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943328:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943331:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943342:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766363.943356:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076510080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766363.943356:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076509952:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766363.943382:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943401:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766363.943404:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943409:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943413:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943417:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943421:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766363.943434:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076510208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766363.943445:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766363.943458:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943462:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766363.943465:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943469:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943473:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943476:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943481:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766363.943489:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076510336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766363.943497:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943501:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766363.943505:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766363.943506:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350930e00 x1862023076509952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766363.943519:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943530:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943534:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766363.943539:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076509952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766363.943551:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766363.943558:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.943568:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766363.943569:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076510464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766363.943579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076509952:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766363.943597:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943601:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766363.943603:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943607:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943610:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943614:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943617:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766363.943618:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766363.943631:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076510592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766363.943643:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766363.943648:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766363.943654:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766363.943656:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943667:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00100000:2.0:1775766363.943668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350930000 x1862023076510080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766363.943681:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766363.943687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076510080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766363.943693:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766363.943708:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766363.943710:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775766363.943713:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.943721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766363.943738:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076510720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766363.943743:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510208:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766363.943769:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943773:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1775766363.943778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766363.943785:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766363.943789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350930a80 x1862023076510208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.943800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076510208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766363.943805:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943809:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766363.943817:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943830:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775766363.943832:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.943838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.943842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766363.943848:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076510848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766363.943854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510336:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766363.943873:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766363.943875:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766363.943878:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766363.943883:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2865 00000100:00080000:1.0:1775766363.943888:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766363.943892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766363.943896:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766363.943902:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766363.943907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350932680 x1862023076510336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766363.943917:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076510976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766363.943926:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076510336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.943943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.943947:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.943950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766363.943956:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766363.943968:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766363.943973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350931880 x1862023076510464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.943987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076510464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.943994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.943996:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.943998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076510464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766363.944011:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766363.944024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766363.944051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350930700 x1862023076510592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.944059:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076510592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.944065:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.944067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.944069:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766363.944073:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766363.944085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766363.944090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350931c00 x1862023076510720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.944097:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076510720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.944111:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.944127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.944130:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766363.944135:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766363.944150:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766363.944158:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350932a00 x1862023076510848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.944165:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076510848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.944170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.944173:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.944175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766363.944178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766363.944188:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766363.944193:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766363/real 1775766363] req@ffffa04350932300 x1862023076510976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766363.944200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076510976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766363.944207:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766363.944211:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766363.944220:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076510976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766364.135165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766364.967133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766365.159145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766365.991128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766365.991293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766366.183169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766367.015139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766367.207166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766368.039162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766368.039241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766368.231120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766369.063161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766369.063217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766369.063220:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766369.063233:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063245:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063255:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063273:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766369.063334:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076511104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766369.063385:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063390:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063394:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063398:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063408:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766369.063422:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076511232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766369.063455:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076511104:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766369.063475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766369.063482:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063487:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063504:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063510:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766369.063526:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076511360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766369.063542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063545:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063549:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063553:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063568:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766369.063579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076511488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063592:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766369.063594:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063598:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063610:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063614:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063621:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766369.063623:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766369.063637:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076511616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766369.063664:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063669:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063673:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063690:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766369.063696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766369.063720:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076511744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775766369.063757:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511488:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766369.063758:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063778:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063784:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063788:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063794:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766369.063810:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076511872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766369.063815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511616:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766369.063848:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766369.063851:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076511744:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766369.063857:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766369.063861:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063878:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063886:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766369.063891:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766369.063896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076511872:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766369.063899:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766369.063919:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076512000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766369.063940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766369.063944:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766369.063946:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766369.063951:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2870 00000100:00080000:0.0:1775766369.063956:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766369.063965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512000:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766369.063976:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766369.063984:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766369.064006:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076512128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766369.064087:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766369.255159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766370.087159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766370.087284:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766370.279115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766371.111191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766371.111328:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766371.303156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766372.135143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766372.135340:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775766372.135432:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775766372.135473:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766372.135505:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135561:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135577:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135588:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766372.135595:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349643800 x1862023076512128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766372.135599:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135613:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135625:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766372.135641:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766372.135654:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076512128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766372.135662:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:2.0:1775766372.135684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766372.135704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349642a00 x1862023076512000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076512000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766372.135745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349641180 x1862023076511872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135758:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076511872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076511872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766372.135781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349641880 x1862023076511616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076511616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135818:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766372.135828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349642300 x1862023076511744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076511744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076511744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766372.135865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349640700 x1862023076511232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076511232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135892:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135896:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766372.135905:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349643b80 x1862023076511360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076511360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135943:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766372.135954:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349640380 x1862023076511488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.135967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076511488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.135976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.135980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.135984:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076511488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766372.135991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766369/real 1775766372] req@ffffa04349643100 x1862023076511104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766372.136005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076511104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766372.136020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766372.136024:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766372.136058:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076511104:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766372.327138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766373.159243:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766373.159313:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766373.351127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766374.183245:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766374.183260:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.183298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1775766374.183303:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766374.183304:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.183332:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.183348:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183356:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766374.183410:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076512256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766374.183453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.183458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766374.183461:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.183466:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.183470:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183493:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.183504:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766374.183525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076512384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766374.183546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766374.183554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076512256:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766374.183556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766374.183562:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.183577:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.183585:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183592:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.183601:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766374.183625:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076512512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766374.183670:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766374.183672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.183680:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766374.183684:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.183695:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.183708:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183715:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.183727:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766374.183746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349641500 x1862023076512256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766374.183755:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076512640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766374.183797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.183803:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775766374.183808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076512256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766374.183820:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766374.183833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766374.183839:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.183847:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183853:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.183861:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766374.183871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.183881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076512256:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766374.183887:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076512768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766374.183909:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.183915:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766374.183918:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.183929:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00100000:2.0:1775766374.183932:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766374.183949:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.183960:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.183971:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766374.183994:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076512896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766374.184013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766374.184091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766374.184109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766374.184113:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766374.184122:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349640e00 x1862023076512384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766374.184126:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.184140:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.184151:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.184161:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766374.184167:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076512384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766374.184186:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076513024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766374.184187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184204:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184220:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766374.184222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.184235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766374.184239:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766374.184242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512512:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766374.184251:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.184258:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766374.184266:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.184276:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766374.184298:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766374.184305:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076513152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766374.184330:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349641c00 x1862023076512512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766374.184346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766374.184358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766374.184391:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766374.184402:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2875 00000100:00080000:0.0:1775766374.184410:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766374.184412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076512512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766374.184423:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766374.184441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775766374.184445:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184455:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184463:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766374.184470:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076513280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766374.184485:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766374.184536:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.184559:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349640a80 x1862023076512640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.184578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076512640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.184591:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184604:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766374.184614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766374.184648:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.184664:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349642680 x1862023076512768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.184682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076512768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.184716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076512768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766374.184761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766374.184801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.184821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349641500 x1862023076512896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.184842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076512896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.184857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184868:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076512896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766374.184879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766374.184912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.184931:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349643100 x1862023076513024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.184951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076513024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.184965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.184971:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.184988:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766374.184998:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766374.185108:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.185127:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349640380 x1862023076513152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.185154:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076513152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.185165:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.185170:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.185175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766374.185221:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766374.185255:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766374.185276:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766374/real 1775766374] req@ffffa04349643b80 x1862023076513280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766374.185300:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076513280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766374.185325:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766374.185330:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766374.185335:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076513280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766374.375103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766375.207173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766375.207448:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766375.399128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766376.231199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766376.231297:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766376.423204:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766377.255177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766377.255311:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766377.447183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766378.279089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766378.279154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766378.471087:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766379.303174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766379.303185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766379.303191:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766379.303194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766379.303204:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303215:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766379.303227:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766379.303254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766379.303271:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076513408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766379.303308:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766379.303312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766379.303328:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766379.303334:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303340:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766379.303343:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076513408:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766379.303349:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766379.303361:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766379.303397:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076513536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766379.303411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766379.303415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766379.303418:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766379.303422:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303427:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303433:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766379.303439:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766379.303445:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766379.303462:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076513664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766379.303463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349640700 x1862023076513408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766379.303505:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076513408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766379.303513:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766379.303523:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:2.0:1775766379.303524:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766379.303531:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766379.303531:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.303539:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076513408:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766379.303541:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766379.303567:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766379.303582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766379.303589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076513792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766379.303602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766379.303608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766379.303610:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766379.303615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349642300 x1862023076513536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303619:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303628:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766379.303633:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076513536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303636:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766379.303647:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766379.303648:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.303656:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.303659:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766379.303666:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076513920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766379.303666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766379.303683:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766379.303688:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303689:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766379.303693:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766379.303699:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00100000:2.0:1775766379.303700:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349641880 x1862023076513664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303704:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766379.303711:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766379.303721:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766379.303727:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076513664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766379.303734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766379.303735:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076514048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766379.303741:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.303747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766379.303759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766379.303763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766379.303766:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766379.303767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513792:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766379.303779:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303785:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766379.303785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303794:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766379.303799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349641180 x1862023076513792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766379.303801:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766379.303814:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076513792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766379.303821:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076514176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766379.303821:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.303828:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.303834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766379.303839:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766379.303853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303863:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766379.303865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349642a00 x1862023076513920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766379.303876:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766379.303883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076513920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303884:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.303893:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766379.303900:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766379.303911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766379.303912:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775766379.303917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.303919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076513920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766379.303925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766379.303931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076514304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766379.303943:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766379.303943:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766379.303949:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766379.303952:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766379.303955:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349643800 x1862023076514048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.303961:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2880 00000100:00080000:0.0:1775766379.304004:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766379.304009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076514048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766379.304012:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766379.304049:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.304053:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766379.304055:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766379.304056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766379.304075:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076514432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766379.304081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766379.304108:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766379.304135:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349641880 x1862023076514176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766379.304144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076514176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766379.304149:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.304152:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.304159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766379.304163:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766379.304180:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766379.304191:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349642300 x1862023076514304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766379.304203:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076514304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766379.304212:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.304215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.304219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766379.304234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766379.304252:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766379.304260:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766379/real 1775766379] req@ffffa04349640700 x1862023076514432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766379.304267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076514432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766379.304273:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766379.304275:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766379.304277:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076514432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766379.495109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766380.327089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766380.327218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766380.519132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766381.351138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766381.351163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766381.543107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766382.375102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766382.375137:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766382.567111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766383.399139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766383.399212:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766383.591134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766384.423194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766384.423233:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423251:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423267:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423280:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766384.423284:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766384.423337:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076514560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766384.423371:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423375:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423378:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423382:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423393:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766384.423404:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076514688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766384.423414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076514560:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766384.423415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766384.423424:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423451:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423455:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423458:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423463:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766384.423473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076514816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766384.423485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423488:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423490:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423494:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423498:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766384.423503:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766384.423508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076514944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766384.423519:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423523:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423528:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423530:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423533:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766384.423539:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076515072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423553:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766384.423555:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423559:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423563:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766384.423565:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349643b80 x1862023076514560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766384.423569:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423578:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766384.423592:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076515200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766384.423592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076514560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766384.423621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423625:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766384.423626:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766384.423628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766384.423630:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423635:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423639:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766384.423639:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766384.423647:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766384.423650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076514560:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766384.423656:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076515328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423665:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766384.423666:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766384.423669:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423677:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423681:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766384.423685:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514688:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766384.423687:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766384.423699:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076515456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766384.423707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766384.423710:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766384.423712:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766384.423717:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766384.423720:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2886 00000100:00080000:0.0:1775766384.423725:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766384.423728:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766384.423731:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766384.423741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349640380 x1862023076514688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766384.423754:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076515584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766384.423762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076514688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.423772:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.423776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.423779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766384.423785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766384.423805:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.423816:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349643100 x1862023076514816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.423834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076514816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.423841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.423845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.423847:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766384.423852:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766384.423876:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.423882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349641500 x1862023076514944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.423890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076514944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.423896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.423897:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.423899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076514944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766384.423902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766384.423913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.423946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349642680 x1862023076515072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.423955:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076515072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.423972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.423975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.423977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766384.423981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766384.423998:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.424005:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349640a80 x1862023076515200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.424012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076515200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.424018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.424019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.424021:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766384.424044:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766384.424058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.424064:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349641c00 x1862023076515328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.424072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076515328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.424078:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.424080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.424090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766384.424118:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766384.424141:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.424153:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349640e00 x1862023076515456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.424162:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076515456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.424168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.424170:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.424172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766384.424175:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766384.424188:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766384.424194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766384/real 1775766384] req@ffffa04349643b80 x1862023076515584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766384.424202:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076515584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766384.424207:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766384.424208:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766384.424210:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076515584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766384.615178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766385.447109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766385.447262:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766385.639145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766386.471137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766386.663161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766387.495181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766387.495242:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766387.687128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766388.519181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766388.519279:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766388.711148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766389.543211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766389.543232:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543249:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543271:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766389.543272:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766389.543284:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766389.543345:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076515712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543371:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766389.543377:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543382:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543387:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543391:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543402:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766389.543420:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076515840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766389.543423:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076515712:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766389.543440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766389.543462:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543467:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543471:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543475:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543481:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766389.543493:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076515968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766389.543506:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543509:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543511:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543520:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766389.543525:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543533:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766389.543546:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076516096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766389.543558:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543561:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543571:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543574:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543585:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766389.543598:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076516224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543607:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775766389.543616:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350932300 x1862023076515712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766389.543617:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543626:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543636:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543648:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543652:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766389.543660:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076516352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543701:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543704:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766389.543706:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543713:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543718:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766389.543723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076515712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766389.543728:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543741:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766389.543760:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076516480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766389.543760:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.543783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766389.543788:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543793:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775766389.543795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076515712:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766389.543812:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543819:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543822:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543826:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543832:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766389.543833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766389.543855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076516608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766389.543877:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766389.543880:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766389.543881:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766389.543885:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2891 00000100:00080000:1.0:1775766389.543888:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766389.543890:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766389.543894:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766389.543907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766389.543908:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076516736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766389.543939:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350932a00 x1862023076515840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.543959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076515840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.543976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.543982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.543988:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766389.544000:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766389.544076:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544098:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350931c00 x1862023076515968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544134:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076515968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544144:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544151:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076515968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766389.544159:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766389.544185:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350930700 x1862023076516096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544210:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076516096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544218:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544225:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766389.544231:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766389.544247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544256:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350931880 x1862023076516224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076516224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544292:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766389.544306:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766389.544334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544350:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350932680 x1862023076516352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544371:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076516352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544387:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544392:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544398:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766389.544409:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766389.544440:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350930a80 x1862023076516480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544467:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076516480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766389.544517:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766389.544541:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350930000 x1862023076516608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544573:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076516608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544586:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544591:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544597:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766389.544608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766389.544642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766389.544661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766389/real 1775766389] req@ffffa04350930e00 x1862023076516736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766389.544678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076516736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766389.544687:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766389.544691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766389.544694:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076516736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766389.735189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766390.567154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766390.567254:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766391.591136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766391.783110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766392.615135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766392.615260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766392.807185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766393.639123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766393.831126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766394.663176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766394.663261:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766394.663563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076516864:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775766394.663678:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.663740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825ce00 x1862023076516864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.663770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076516864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.663864:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.663879:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.663888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076516864:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766394.663930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766394.663976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.664002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825d880 x1862023076516992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.664083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076516992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.664099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.664106:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.664111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076516992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766394.664122:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766394.664160:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.664183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825dc00 x1862023076517120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.664213:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076517120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.664224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.664233:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.664237:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766394.664367:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766394.664440:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.664497:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825d500 x1862023076517248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.664554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076517248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.664596:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.664607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.664618:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766394.664660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766394.664699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.664726:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825f100 x1862023076517376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.664766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076517376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.664782:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.664798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.664811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076517376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766394.664858:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766394.664901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.664921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825e300 x1862023076517504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.664965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076517504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.664993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.665002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.665011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766394.665093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766394.665133:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.665171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825ed80 x1862023076517632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.665187:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076517632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.665198:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.665202:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.665207:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766394.665214:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766394.665233:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.665244:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825c000 x1862023076517760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.665257:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076517760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.665285:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.665290:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.665294:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766394.665301:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766394.665327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766394.665338:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766394/real 1775766394] req@ffffa0435825e680 x1862023076517888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766394.665352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076517888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766394.665361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766394.665365:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766394.665368:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076517888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766394.855159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766395.687154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766395.879193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766396.711218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766396.903148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766397.735176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766397.735214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766397.927192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766398.759200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766398.759292:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766398.951199:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766399.783150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766399.783160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766399.783175:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783198:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.783211:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783218:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766399.783219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766399.783258:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076518016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766399.783300:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783304:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766399.783306:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783310:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.783314:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783318:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766399.783328:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766399.783341:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076518144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766399.783354:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076518016:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766399.783382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766399.783387:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783394:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.783426:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783431:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766399.783438:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766399.783451:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076518272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766399.783461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775766399.783472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766399.783473:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783480:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.783485:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783488:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766399.783495:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766399.783508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076518400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766399.783517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766399.783523:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783528:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00100000:1.0:1775766399.783536:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7f480 x1862023076518016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766399.783542:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783581:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766399.783588:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076518016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766399.783591:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766399.783630:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766399.783638:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076518528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766399.783641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.783652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076518016:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766399.783665:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.783683:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766399.783686:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783692:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.783697:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.783702:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766399.783710:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766399.783713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766399.783745:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076518656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766399.783749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.783791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7dc00 x1862023076518144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766399.783817:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766399.783821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076518144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766399.783825:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766399.783834:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.783855:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:1.0:1775766399.783857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.783867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766399.783868:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766399.783873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766399.783882:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766399.783886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518272:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766399.783896:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766399.783909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.783920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7c000 x1862023076518272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766399.783925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076518784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766399.783943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076518272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.783957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.783973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.783983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766399.783990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518400:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766399.784020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766399.784023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766399.784052:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.784058:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2901 00000800:00000100:1.0:1775766399.784060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766399.784071:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.784079:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766399.784081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7f100 x1862023076518400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766399.784088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766399.784118:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076518400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766399.784121:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076518912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766399.784141:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784163:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518400:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766399.784171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766399.784192:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518528:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766399.784194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766399.784207:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766399.784221:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2901 00000100:00080000:2.0:1775766399.784229:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766399.784238:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766399.784245:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766399.784262:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766399.784268:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7e300 x1862023076518528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766399.784305:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076519040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766399.784306:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076518528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.784344:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784353:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784363:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076518528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766399.784396:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766399.784412:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.784419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7f800 x1862023076518656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766399.784427:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076518656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.784444:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784447:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784449:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766399.784454:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766399.784468:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.784475:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7c700 x1862023076518784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766399.784483:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076518784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.784489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784491:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766399.784497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766399.784507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.784523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7f480 x1862023076518912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766399.784531:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076518912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.784536:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784538:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076518912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766399.784549:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766399.784563:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766399.784570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766399/real 1775766399] req@ffffa0434aa7e680 x1862023076519040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766399.784578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076519040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766399.784584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766399.784586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766399.784587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766399.975149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766400.807156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766400.807281:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766400.999121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766401.831101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766401.831133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766402.023117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766402.855207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766402.855358:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766403.047144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766403.879127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766403.879170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766404.071190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766404.903180:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766404.903188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766404.903190:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766404.903211:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903222:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903235:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903242:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766404.903254:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766404.903287:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076519168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766404.903317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903321:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903323:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903327:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903336:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766404.903347:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076519296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903372:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766404.903378:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076519168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766404.903379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903390:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903402:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903405:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903409:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766404.903419:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076519424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903437:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903440:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766404.903442:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903445:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903449:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903452:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903456:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766404.903463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076519552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903468:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903471:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766404.903473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903477:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903479:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903481:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766404.903500:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076519680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903506:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903509:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766404.903510:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903512:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903519:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766404.903529:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766404.903535:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766404.903553:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076519808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903560:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903563:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766404.903566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903570:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903579:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903582:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766404.903594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076519936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903602:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903605:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766404.903606:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903610:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903614:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903617:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766404.903621:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766404.903629:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7ea00 x1862023076519168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766404.903652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076520064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766404.903758:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766404.903765:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766404.903767:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766404.903771:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2906 00000100:00080000:2.0:1775766404.903777:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766404.903780:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766404.903786:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076519168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766404.903787:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766404.903810:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076520192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766404.903813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.903826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.903837:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076519168:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766404.903898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766404.903934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.903950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7d500 x1862023076519296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.903963:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076519296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.903972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.903975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.903978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766404.903985:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766404.904003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904011:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7d880 x1862023076519424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076519424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904087:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904092:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904096:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766404.904106:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766404.904135:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904149:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7ce00 x1862023076519552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904177:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076519552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904188:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904192:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904196:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766404.904204:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766404.904243:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904251:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7ed80 x1862023076519680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076519680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904270:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904272:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904274:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076519680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766404.904279:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766404.904294:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7fb80 x1862023076519808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076519808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904329:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904333:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904335:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766404.904341:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766404.904359:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7c380 x1862023076519936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076519936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904403:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904408:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904412:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076519936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766404.904421:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766404.904446:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904458:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7df80 x1862023076520064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904469:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076520064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904478:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904506:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766404.904515:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766404.904542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766404.904554:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766404/real 1775766404] req@ffffa0434aa7e680 x1862023076520192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766404.904568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076520192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766404.904579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766404.904582:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766404.904586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766405.095214:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766405.927139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766405.927208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766406.119123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766406.951176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766406.951207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766407.143175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766407.975094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766407.975141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766408.167129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766408.999098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766408.999145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766409.191101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766410.023168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766410.023228:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.023235:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766410.023240:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023252:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023265:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023274:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766410.023306:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766410.023329:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076520320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.023403:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766410.023422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766410.023429:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076520320:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766410.023441:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023452:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023459:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.023474:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766410.023505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076520448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023527:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.023535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1775766410.023539:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766410.023542:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023553:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023562:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023570:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.023582:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766410.023595:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7f480 x1862023076520320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766410.023610:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076520576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023633:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.023640:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775766410.023644:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076520320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766410.023646:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023689:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:1.0:1775766410.023702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766410.023703:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023716:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766410.023721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.023733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076520320:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766410.023739:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766410.023762:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076520704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023778:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766410.023784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520448:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766410.023785:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766410.023788:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023794:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023801:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023808:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.023815:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766410.023833:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076520832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023846:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766410.023851:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766410.023855:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766410.023860:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023867:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023872:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023877:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.023883:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766410.023886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7c700 x1862023076520448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766410.023900:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076520960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.023927:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.023931:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766410.023939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076520448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766410.023942:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.023956:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.023966:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.023976:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766410.023980:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766410.023983:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766410.023992:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766410.024003:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076521088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766410.024004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520448:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766410.024074:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766410.024079:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775766410.024082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520576:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766410.024093:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.024109:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.024113:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766410.024118:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.024126:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766410.024142:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076521216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766410.024155:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766410.024156:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766410.024164:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766410.024167:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766410.024175:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2911 00000100:00080000:2.0:1775766410.024181:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766410.024188:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7f800 x1862023076520576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766410.024189:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766410.024201:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766410.024214:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076520576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024229:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766410.024234:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076521344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766410.024240:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024253:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766410.024289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766410.024317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024329:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7e300 x1862023076520704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024363:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076520704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024373:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024377:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024383:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766410.024395:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766410.024430:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024450:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7f100 x1862023076520832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076520832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024495:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024500:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076520832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766410.024508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766410.024535:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7c000 x1862023076520960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076520960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024586:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024594:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076520960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766410.024601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766410.024623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024642:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7dc00 x1862023076521088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024655:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076521088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024693:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766410.024715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766410.024752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024773:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7f480 x1862023076521216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076521216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766410.024829:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766410.024856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766410.024869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766410/real 1775766410] req@ffffa0434aa7e680 x1862023076521344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766410.024890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076521344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766410.024905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766410.024910:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766410.024915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076521344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766410.215160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766411.047114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766411.047347:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766411.239170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766412.071117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766412.071176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766412.263117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766413.095150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766413.095178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766413.287113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766414.119169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766414.119410:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766414.311153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766415.143215:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766415.143225:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766415.143232:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766415.143242:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.143262:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00080000:0.0:1775766415.143281:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775766415.143289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766415.143296:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766415.143382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076521472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766415.143433:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.143441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766415.143445:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.143455:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00080000:0.0:1775766415.143465:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766415.143476:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766415.143493:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076521472:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766415.143495:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766415.143529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076521600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766415.143554:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.143561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766415.143566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.143575:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2916 00000800:00000100:1.0:1775766415.143584:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766415.143588:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766415.143632:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766415.143647:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766415.143672:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076521728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766415.143692:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349640700 x1862023076521472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.143697:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.143713:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766415.143722:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.143746:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00100000:1.0:1775766415.143751:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076521472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.143758:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766415.143772:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766415.143784:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766415.143798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766415.143809:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076521856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766415.143813:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.143831:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076521472:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766415.143851:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.143858:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766415.143862:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.143872:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00080000:0.0:1775766415.143880:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766415.143889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521600:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766415.143893:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766415.143904:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766415.143934:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076521984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766415.143954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766415.143960:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.143969:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766415.143974:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766415.143979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349642300 x1862023076521600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.143988:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00080000:0.0:1775766415.144006:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766415.144015:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076521600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.144019:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766415.144073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.144080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.144086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766415.144096:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766415.144125:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766415.144142:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766415.144152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349641880 x1862023076521728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766415.144172:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076521728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766415.144181:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076522112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766415.144187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.144193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.144203:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766415.144222:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521856:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766415.144238:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.144246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766415.144251:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.144261:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2916 00000800:00000100:1.0:1775766415.144271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766415.144274:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766415.144288:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766415.144300:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766415.144301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349643800 x1862023076521856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766415.144334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076522240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766415.144398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766415.144406:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076521856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.144409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766415.144419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766415.144433:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766415.144438:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00080000:1.0:1775766415.144444:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.144453:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521856:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766415.144454:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766415.144469:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766415.144479:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766415.144491:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766415.144507:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076522368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766415.144529:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766415.144533:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766415.144542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775766415.144548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349642a00 x1862023076521984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.144550:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766415.144569:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2916 00000100:00100000:1.0:1775766415.144579:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076521984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766415.144599:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766415.144600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.144641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766415.144646:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766415.144657:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076521984:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775766415.144662:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766415.144673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766415.144695:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076522496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766415.144725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766415.144749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349641180 x1862023076522112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766415.144766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076522112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766415.144805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.144812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.144818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766415.144830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766415.144872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766415.144894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349640700 x1862023076522240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766415.144913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076522240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766415.144923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.144927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.144931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766415.144953:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766415.144978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766415.144990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349643b80 x1862023076522368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766415.145004:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076522368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766415.145013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.145017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.145062:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766415.145071:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766415.145099:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766415.145113:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766415/real 1775766415] req@ffffa04349640e00 x1862023076522496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766415.145127:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076522496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766415.145137:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766415.145141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766415.145144:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076522496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766415.335115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766416.167110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766416.167246:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766416.359106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766417.191168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766417.191212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766417.383108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766418.215107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766418.215165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766418.407126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766419.239108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766419.239146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766419.431128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766420.263125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766420.263127:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263135:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766420.263156:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263165:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263174:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263181:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766420.263221:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076522624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766420.263243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766420.263248:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775766420.263250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766420.263251:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263256:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263260:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076522624:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766420.263286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766420.263307:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076522752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766420.263321:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263325:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766420.263327:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263332:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263337:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263340:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766420.263344:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263347:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766420.263360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076522880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766420.263368:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263371:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766420.263372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263376:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263379:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766420.263383:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7df80 x1862023076522624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766420.263385:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263392:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766420.263405:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076523008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766420.263408:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076522624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766420.263433:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766420.263438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766420.263440:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775766420.263442:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076522624:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766420.263449:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263456:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00100000:0.0:1775766420.263462:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076522752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766420.263462:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263471:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775766420.263484:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766420.263493:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076523136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766420.263496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7c380 x1862023076522752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766420.263503:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263508:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766420.263509:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263512:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263515:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263521:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263528:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766420.263536:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076522752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766420.263540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076523264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766420.263567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766420.263594:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766420.263598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076522752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766420.263600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766420.263605:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766420.263609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076522880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766420.263612:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263620:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263624:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263629:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766420.263640:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766420.263651:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076523392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766420.263655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7fb80 x1862023076522880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076522880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766420.263701:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766420.263704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766420.263705:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766420.263708:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766420.263711:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766420.263715:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00100000:0.0:1775766420.263719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076522880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766420.263721:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263727:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523008:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766420.263732:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766420.263744:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076523520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766420.263751:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766420.263753:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766420.263754:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766420.263757:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2921 00000100:00080000:2.0:1775766420.263759:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766420.263761:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766420.263765:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766420.263774:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076523648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766420.263775:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7ed80 x1862023076523008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076523008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263819:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.263823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766420.263837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766420.263852:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263859:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7ce00 x1862023076523136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076523136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.263876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766420.263879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766420.263889:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7d880 x1862023076523264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263901:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076523264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263908:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.263916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766420.263920:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766420.263932:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7d500 x1862023076523392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263945:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076523392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263952:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.263954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766420.263957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766420.263967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.263972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7ea00 x1862023076523520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.263987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076523520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.263992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.263994:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.263996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766420.264000:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766420.264010:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766420.264022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766420/real 1775766420] req@ffffa0434aa7df80 x1862023076523648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766420.264047:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076523648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766420.264053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766420.264055:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766420.264057:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076523648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766420.455091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766421.287151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766421.287196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766421.479185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766422.311149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766422.311208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766422.503109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775766423.335196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766423.335200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766423.527149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766424.359112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766424.359203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766424.551219:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766425.383249:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383258:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766425.383264:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383279:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00100000:1.0:1775766425.383288:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766425.383291:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383300:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766425.383342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076523776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775766425.383354:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766425.383376:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383380:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766425.383382:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383385:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00080000:2.0:1775766425.383389:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383392:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383401:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766425.383412:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076523904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766425.383420:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076523776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766425.383423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383431:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766425.383433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383436:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00080000:2.0:1775766425.383439:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383441:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766425.383472:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076524032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766425.383480:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383482:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766425.383491:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383497:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2927 00000800:00000100:1.0:1775766425.383510:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766425.383518:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383523:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766425.383542:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076524160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766425.383551:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766425.383554:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7e680 x1862023076523776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766425.383555:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766425.383558:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383568:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00080000:2.0:1775766425.383574:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383578:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383582:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766425.383586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076523776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766425.383593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076524288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766425.383601:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383611:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766425.383615:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766425.383621:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.383628:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766425.383631:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00100000:1.0:1775766425.383634:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076523776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766425.383637:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383644:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383649:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766425.383661:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766425.383665:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076524416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766425.383690:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383695:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1775766425.383699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766425.383700:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383708:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00100000:1.0:1775766425.383714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7f480 x1862023076523904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766425.383714:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383725:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766425.383733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076523904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766425.383733:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766425.383745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.383751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766425.383751:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076524544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766425.383756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076523904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766425.383769:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766425.383773:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766425.383776:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766425.383778:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383783:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00080000:2.0:1775766425.383788:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766425.383792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766425.383796:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766425.383801:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766425.383805:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7dc00 x1862023076524032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766425.383814:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076524672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766425.383818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076524032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.383839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.383846:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766425.383848:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766425.383851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766425.383852:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766425.383854:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766425.383858:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2927 00000100:00080000:2.0:1775766425.383861:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766425.383867:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766425.383870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524160:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766425.383871:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766425.383889:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076524800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766425.383900:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.383910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7c000 x1862023076524160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.383920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076524160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.383929:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.383933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.383936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766425.383943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766425.383964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.383977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7f100 x1862023076524288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.383992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076524288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.384003:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.384006:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.384009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076524288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766425.384074:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766425.384132:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.384157:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7e300 x1862023076524416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.384197:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076524416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.384203:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.384206:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.384208:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766425.384213:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766425.384236:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.384249:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7f800 x1862023076524544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.384263:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076524544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.384273:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.384277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.384281:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766425.384288:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766425.384307:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.384316:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7c700 x1862023076524672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.384325:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076524672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.384341:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.384344:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.384346:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766425.384351:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766425.384366:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766425.384375:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766425/real 1775766425] req@ffffa0434aa7e680 x1862023076524800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766425.384387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076524800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766425.384397:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766425.384401:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766425.384405:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076524800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766425.575225:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766426.407091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766426.407155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766426.599114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766427.431165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766427.431259:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766427.623114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766428.455108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766428.455156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766429.479106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766429.479191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766429.671115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766430.503154:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503163:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766430.503168:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503176:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503193:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766430.503195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766430.503231:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076524928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503253:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503257:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766430.503273:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503277:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503281:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503285:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503294:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766430.503309:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076525056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503324:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766430.503326:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775766430.503330:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766430.503331:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503336:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503340:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503344:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766430.503354:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076525184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503362:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503368:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766430.503372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503375:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503379:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503382:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766430.503401:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076525312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503410:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766430.503422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503431:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503435:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503439:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503466:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766430.503476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076525440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503483:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503485:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766430.503486:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503489:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503491:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766430.503502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076525568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503511:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766430.503512:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503521:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503524:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503527:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503530:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766430.503537:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076525696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503544:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503546:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766430.503547:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503549:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503551:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503553:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503556:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766430.503561:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076525824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766430.503566:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766430.503568:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766430.503570:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766430.503572:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2932 00000100:00080000:2.0:1775766430.503574:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766430.503576:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766430.503579:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766430.503585:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076525952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766430.695152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766431.527137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766431.527243:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766431.719137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766432.551183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766432.551235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766432.743160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766433.575157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766433.575289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766433.767172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766434.599181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766434.599276:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766434.791111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766435.623146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766435.623150:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766435.623164:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623174:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623184:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623191:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766435.623215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766435.623231:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076526080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623257:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623260:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766435.623261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623264:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623267:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623270:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623278:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766435.623289:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076526208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623329:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623331:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766435.623333:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623337:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623341:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623344:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766435.623370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076526336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623378:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623381:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766435.623383:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623388:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623438:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623445:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623451:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766435.623467:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076526464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623478:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623498:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766435.623501:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623505:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623521:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623531:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623539:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766435.623570:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076526592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623585:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623589:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766435.623591:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623595:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623607:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623630:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623636:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766435.623651:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076526720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623661:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623664:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766435.623665:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623669:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623673:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623678:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623681:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766435.623691:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076526848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623711:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623714:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766435.623715:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623722:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623726:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623732:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623738:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766435.623748:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076526976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766435.623755:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766435.623758:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766435.623760:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766435.623765:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2937 00000100:00080000:2.0:1775766435.623770:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766435.623777:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766435.623786:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766435.623803:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076527104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766435.815110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766436.647116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766436.647226:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766436.839126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766437.671348:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766437.863125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766438.695151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766438.695187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766438.695480:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7c380 x1862023076527104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076527104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076527104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766438.695592:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7e680 x1862023076526976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695607:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076526976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695618:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076526976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766438.695628:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7f480 x1862023076526848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695641:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076526848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695650:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695653:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695656:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076526848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766438.695663:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7dc00 x1862023076526720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076526720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076526720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766438.695703:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7c000 x1862023076526592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076526592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076526592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766438.695746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7f100 x1862023076526464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076526464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076526464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766438.695771:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7f800 x1862023076526208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695778:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076526208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695784:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076526208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766438.695807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7e300 x1862023076526336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076526336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695835:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076526336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766438.695845:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766435/real 1775766438] req@ffffa0434aa7c700 x1862023076526080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766438.695858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076526080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766438.695866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766438.695870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766438.695873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076526080:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766438.887140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766439.719115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766439.719178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766439.911117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766440.743165:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766440.743169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766440.743178:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766440.743186:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.743198:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.743211:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.743220:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766440.743265:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076527232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766440.743303:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766440.743309:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766440.743312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.743318:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.743326:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.743331:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766440.743362:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766440.743369:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076527232:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766440.743388:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076527360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766440.743407:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766440.743412:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766440.743415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.743421:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.743428:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.743452:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766440.743462:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766440.743463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766440.743490:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076527488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766440.743517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7fb80 x1862023076527232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.743518:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766440.743542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766440.743545:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766440.743553:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076527232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.743558:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.743576:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.743588:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766440.743610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766440.743612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766440.743619:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.743624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076527232:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766440.743643:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076527616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766440.743663:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766440.743664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766440.743678:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766440.743682:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.743690:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.743701:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.743707:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766440.743715:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775766440.743725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766440.743749:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076527744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766440.743752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7ed80 x1862023076527360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.743812:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766440.743817:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076527360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.743841:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766440.743847:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766440.743849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.743864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766440.743869:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00100000:1.0:1775766440.743876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766440.743881:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766440.743890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766440.743893:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766440.743915:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766440.743939:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076527872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766440.743946:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766440.743968:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766440.743971:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7ce00 x1862023076527488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.743978:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766440.743984:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.743994:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.744009:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.744018:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076527488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766440.744066:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766440.744085:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766440.744087:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076528000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766440.744101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766440.744118:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766440.744127:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766440.744133:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.744145:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.744152:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766440.744158:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527616:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766440.744186:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766440.744213:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076528128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766440.744219:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744244:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7d880 x1862023076527616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.744245:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766440.744261:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766440.744266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766440.744286:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2942 00000100:00080000:2.0:1775766440.744295:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766440.744296:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076527616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766440.744310:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766440.744325:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766440.744328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766440.744335:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766440.744362:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766440.744383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076528256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766440.744401:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744429:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7d500 x1862023076527744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766440.744448:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076527744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766440.744457:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.744461:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744465:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076527744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766440.744472:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076527872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766440.744494:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744508:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7ea00 x1862023076527872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766440.744533:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076527872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766440.744542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.744546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076527872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766440.744555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766440.744578:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744589:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7df80 x1862023076528000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766440.744601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076528000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766440.744608:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.744612:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766440.744635:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766440.744654:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7ce00 x1862023076528128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766440.744678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076528128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766440.744699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.744703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744707:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766440.744714:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766440.744736:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766440.744748:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766440/real 1775766440] req@ffffa0434aa7ed80 x1862023076528256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766440.744759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076528256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766440.744769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766440.744773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766440.744777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076528256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766440.935105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766441.767149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766441.767282:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766441.959110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766442.791157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766442.983086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766443.815181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766443.815367:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766444.007136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766444.839172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766445.031145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766445.863127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766445.863184:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863191:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766445.863196:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863210:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2947 00010000:00010000:1.0:1775766445.863210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766445.863224:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863231:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766445.863278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076528384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766445.863342:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863345:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766445.863347:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863351:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863354:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863359:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766445.863378:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076528384:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766445.863383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076528512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766445.863392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863395:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766445.863396:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863400:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863447:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766445.863452:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766445.863456:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766445.863480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076528640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766445.863493:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766445.863496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349641c00 x1862023076528384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766445.863500:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766445.863504:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863513:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863518:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863522:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766445.863526:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076528384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766445.863528:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766445.863541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076528768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766445.863544:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.863553:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766445.863555:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863558:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766445.863560:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766445.863560:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076528384:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766445.863568:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863571:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863577:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766445.863584:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076528896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766445.863601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528512:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766445.863616:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863619:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766445.863621:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863625:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2947 00000800:00000100:1.0:1775766445.863628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766445.863634:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863639:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863645:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766445.863645:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349640a80 x1862023076528512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.863661:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076528512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766445.863663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076529024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766445.863683:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766445.863688:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766445.863690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.863696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528512:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766445.863698:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766445.863703:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766445.863705:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528640:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766445.863712:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863718:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766445.863725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766445.863727:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863735:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766445.863735:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349642680 x1862023076528640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.863748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076528640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766445.863754:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076529152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766445.863760:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.863767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766445.863770:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766445.863772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528640:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766445.863774:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766445.863776:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863783:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00100000:1.0:1775766445.863799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528768:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766445.863807:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766445.863812:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863818:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766445.863828:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076529280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766445.863836:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766445.863843:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766445.863846:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766445.863848:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766445.863852:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2947 00000100:00080000:0.0:1775766445.863856:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766445.863857:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349641500 x1862023076528768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766445.863861:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766445.863867:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766445.863880:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076529408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766445.863884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076528768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.863904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.863908:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.863912:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766445.863934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766445.863954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766445.863964:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349643100 x1862023076528896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.863975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076528896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.863983:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.863987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.863990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076528896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766445.863996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766445.864012:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766445.864021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349640380 x1862023076529024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.864065:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076529024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.864074:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.864077:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.864080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766445.864087:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766445.864107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766445.864117:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349641c00 x1862023076529152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.864129:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076529152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.864137:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.864140:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.864143:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766445.864149:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766445.864165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766445.864174:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349640e00 x1862023076529280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.864185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076529280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.864205:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.864208:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.864212:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766445.864218:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766445.864237:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766445.864248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766445/real 1775766445] req@ffffa04349643b80 x1862023076529408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766445.864260:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076529408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766445.864266:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766445.864269:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766445.864271:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076529408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766446.055104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766446.887150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766447.079103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766447.911120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766447.911229:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766448.103103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766448.935141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766449.127095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766449.959098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766449.959338:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766450.151140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766450.983124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766450.983145:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1775766450.983157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766450.983158:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983169:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983178:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983183:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766450.983217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076529536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766450.983273:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983276:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766450.983277:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983281:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983284:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983288:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983295:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766450.983306:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076529664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766450.983314:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983319:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766450.983320:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076529536:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766450.983320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983327:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983329:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983331:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983334:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766450.983344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076529792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766450.983357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983360:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766450.983361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983375:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983378:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983382:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766450.983391:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076529920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766450.983400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766450.983401:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983404:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766450.983406:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983409:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983412:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983415:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983418:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766450.983425:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076530048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766450.983432:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766450.983435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983438:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983440:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983442:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983445:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766450.983452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7fb80 x1862023076529536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766450.983456:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076530176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766450.983500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766450.983506:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076529536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766450.983507:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983513:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983517:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983523:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766450.983530:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983537:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766450.983538:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076530304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766450.983546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076529536:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766450.983548:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983556:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766450.983558:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983561:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983563:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983565:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983567:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766450.983573:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076530432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766450.983580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766450.983580:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766450.983586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766450.983590:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766450.983603:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2952 00000100:00080000:2.0:1775766450.983611:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766450.983614:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766450.983618:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766450.983626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766450.983629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076530560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766450.983645:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7c700 x1862023076529664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.983660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076529664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.983670:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983674:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.983677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766450.983684:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766450.983703:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.983715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7e300 x1862023076529792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.983742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076529792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.983751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.983758:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766450.983764:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766450.983788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.983799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7f800 x1862023076529920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.983812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076529920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.983820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.983827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076529920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766450.983833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766450.983853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.983864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7f100 x1862023076530048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.983876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076530048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.983899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.983906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766450.983912:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766450.983936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.983960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7c000 x1862023076530176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.983974:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076530176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.983983:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.983986:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.983990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766450.983996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766450.984016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.984050:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7dc00 x1862023076530304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.984064:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076530304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.984073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.984076:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.984087:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766450.984109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766450.984132:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.984143:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7f480 x1862023076530432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.984157:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076530432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.984165:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.984169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.984172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766450.984178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766450.984195:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766450.984205:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766450/real 1775766450] req@ffffa0434aa7e680 x1862023076530560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766450.984225:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076530560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766450.984249:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766450.984253:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766450.984257:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076530560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766451.175100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766452.007142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766452.007294:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766452.199112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766453.031137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766453.031187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766453.223090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766454.055152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766454.055194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766454.247120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766455.079144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766455.079285:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766455.271150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766456.103119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766456.103132:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103139:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766456.103143:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103152:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103162:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775766456.103166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766456.103171:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766456.103226:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076530688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103251:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103254:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766456.103256:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103259:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103262:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766456.103266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103275:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766456.103289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076530688:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766456.103290:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076530816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103325:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766456.103327:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103330:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103334:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766456.103337:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103342:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766456.103351:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766456.103355:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076530944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766456.103368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103370:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103373:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766456.103376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766456.103389:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7c380 x1862023076530688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766456.103392:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076531072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103430:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103433:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766456.103434:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103437:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00100000:1.0:1775766456.103442:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076530688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766456.103449:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766456.103458:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766456.103461:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766456.103463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766456.103469:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.103473:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076530688:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766456.103476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076531200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103486:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103489:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766456.103491:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103493:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103496:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766456.103501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766456.103502:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766456.103522:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076531328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103532:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766456.103536:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766456.103538:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766456.103541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103546:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103551:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766456.103554:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7e680 x1862023076530816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766456.103558:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103565:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766456.103577:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076530816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766456.103581:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076531456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766456.103599:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.103617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766456.103620:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766456.103625:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766456.103626:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766456.103629:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766456.103639:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103642:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766456.103643:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766456.103648:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103655:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766456.103666:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076531584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766456.103690:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766456.103694:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766456.103696:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766456.103701:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766456.103705:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2957 00000100:00080000:2.0:1775766456.103712:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766456.103716:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766456.103721:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766456.103724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7f480 x1862023076530944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766456.103734:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076531712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766456.103741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076530944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.103757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.103761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.103764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076530944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766456.103772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766456.103790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.103798:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7dc00 x1862023076531072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.103815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076531072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.103839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.103842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.103844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766456.103848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766456.103875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.103887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7c000 x1862023076531200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.103912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076531200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.103923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.103926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.103930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766456.103948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766456.103965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.103974:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7f100 x1862023076531328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.103982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076531328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.103989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.103991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.103993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766456.103997:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766456.104008:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.104014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7f800 x1862023076531456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.104036:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076531456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.104042:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.104044:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.104046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766456.104050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766456.104084:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.104093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7e300 x1862023076531584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.104101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076531584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.104107:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.104110:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.104112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766456.104116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766456.104127:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766456.104133:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766456/real 1775766456] req@ffffa0434aa7c700 x1862023076531712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766456.104141:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076531712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766456.104156:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766456.104159:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766456.104161:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076531712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766456.295112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766457.127136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766457.127153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766457.319129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766458.151236:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766458.151322:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766458.343252:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766459.175101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766459.175211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766459.367090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766460.199187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766460.199262:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766460.391163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766461.223159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766461.223180:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223211:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766461.223215:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775766461.223227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766461.223232:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223246:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223255:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766461.223299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076531840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766461.223329:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223332:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766461.223334:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223337:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223340:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223343:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223352:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766461.223361:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076531968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766461.223370:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223373:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766461.223375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223378:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223380:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223389:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766461.223390:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076531840:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766461.223395:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766461.223407:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076532096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766461.223420:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223423:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766461.223425:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223429:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223435:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223439:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766461.223454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076532224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766461.223463:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766461.223473:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223476:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766461.223478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223482:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223485:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223488:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223491:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766461.223514:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076532352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766461.223521:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766461.223526:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7fb80 x1862023076531840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766461.223526:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766461.223534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223553:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223564:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223579:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223584:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766461.223591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076531840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766461.223602:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076532480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766461.223604:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.223614:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.223629:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076531840:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766461.223646:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223651:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766461.223654:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766461.223664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766461.223664:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223676:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223683:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223688:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766461.223695:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766461.223702:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076532608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766461.223720:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223724:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766461.223726:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223731:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00100000:1.0:1775766461.223736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7ed80 x1862023076531968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766461.223738:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766461.223746:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223754:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766461.223765:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076532736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766461.223771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076531968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.223805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.223819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.223836:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076531968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766461.223836:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766461.223848:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766461.223854:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766461.223859:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2962 00000100:00080000:2.0:1775766461.223865:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766461.223866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766461.223874:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766461.223885:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766461.223899:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076532864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766461.223916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.223932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7ce00 x1862023076532096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.223944:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076532096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.223950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.223953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.223956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766461.223962:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766461.223977:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.223983:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7df80 x1862023076532224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.223990:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076532224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766461.224010:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766461.224023:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.224060:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7ea00 x1862023076532352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.224073:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076532352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224083:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224087:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076532352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766461.224097:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766461.224116:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.224139:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7d500 x1862023076532480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.224149:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076532480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224154:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224157:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224164:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766461.224181:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766461.224196:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.224203:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7d880 x1862023076532608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.224211:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076532608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224216:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224218:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224220:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766461.224224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766461.224233:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.224238:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7fb80 x1862023076532736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.224245:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076532736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224250:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224251:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224253:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766461.224257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766461.224265:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766461.224278:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766461/real 1775766461] req@ffffa0434aa7c700 x1862023076532864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766461.224285:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076532864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766461.224291:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766461.224293:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766461.224295:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076532864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766461.415088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766462.247204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766462.247235:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766462.439102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766463.271154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766463.271212:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766463.463175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766464.295187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766464.295240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766464.487164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766465.319163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766465.319231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766465.511180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766466.343158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766466.343174:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343181:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766466.343185:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343219:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343228:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766466.343279:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076532992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343300:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343303:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766466.343304:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343306:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343309:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343312:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766466.343329:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076533120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766466.343340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076532992:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766466.343341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766466.343344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343349:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343351:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343354:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766466.343365:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076533248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343372:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343379:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766466.343380:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343383:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343388:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343391:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343395:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766466.343398:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766466.343406:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076533376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343414:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343417:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766466.343418:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343421:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343423:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343426:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343430:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766466.343433:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349640700 x1862023076532992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766466.343441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076533504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343451:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343455:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775766466.343456:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076532992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766466.343458:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343467:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343473:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766466.343475:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766466.343478:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766466.343481:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766466.343486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766466.343494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076532992:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766466.343501:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076533632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766466.343508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343510:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766466.343512:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766466.343516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766466.343517:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00080000:0.0:1775766466.343522:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343525:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766466.343538:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076533760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766466.343544:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766466.343552:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343555:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766466.343556:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343561:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2968 00000100:00100000:1.0:1775766466.343561:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349641180 x1862023076533120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766466.343566:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343572:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343578:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766466.343581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076533120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766466.343589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076533888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766466.343590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343605:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766466.343606:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766466.343613:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775766466.343616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766466.343621:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766466.343626:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2968 00000800:00000100:1.0:1775766466.343636:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766466.343637:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766466.343642:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766466.343647:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766466.343651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349642a00 x1862023076533248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766466.343658:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076534016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766466.343675:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076533248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766466.343701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766466.343718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.343731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349643800 x1862023076533376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.343741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076533376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343754:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766466.343759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766466.343778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.343786:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349641880 x1862023076533504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.343794:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076533504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343800:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076533504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766466.343807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766466.343826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.343841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349642300 x1862023076533632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.343851:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076533632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766466.343898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766466.343913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.343919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349640700 x1862023076533760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.343927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076533760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343932:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343934:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766466.343939:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766466.343948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.343953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349643b80 x1862023076533888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.343960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076533888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.343964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.343966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.343972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076533888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766466.343976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076534016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766466.343998:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766466.344005:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766466/real 1775766466] req@ffffa04349640e00 x1862023076534016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766466.344014:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076534016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766466.344019:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766466.344021:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766466.344023:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076534016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766466.535123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766467.367179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766467.367203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766467.559192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766468.391156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766468.583176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766469.415193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766469.415267:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766469.607216:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766470.439170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766470.631167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766471.463168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766471.463185:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766471.463192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766471.463197:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463211:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463221:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766471.463248:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766471.463289:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076534144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766471.463323:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463326:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463334:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766471.463364:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076534272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463393:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766471.463397:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463403:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463408:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463413:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463420:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766471.463431:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076534400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766471.463463:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463479:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463486:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463501:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463510:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766471.463525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076534528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766471.463548:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463560:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463568:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463572:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463579:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766471.463592:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076534656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463618:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766471.463620:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463625:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463629:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463638:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463651:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766471.463667:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076534784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766471.463714:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463722:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463734:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463744:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463749:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766471.463770:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076534912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463779:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766471.463792:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463799:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463802:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463811:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463824:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766471.463838:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076535040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766471.463866:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766471.463873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766471.463879:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766471.463890:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2973 00000100:00080000:1.0:1775766471.463898:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766471.463903:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766471.463908:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766471.463925:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076535168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766471.655155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766472.487149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766472.679167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766473.511167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766473.511248:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766473.703121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766474.535147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766474.727185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766475.559188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766475.559256:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766475.751172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766476.583173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766476.583194:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766476.583196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766476.583224:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583238:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583246:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766476.583288:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076535296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583326:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766476.583329:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583335:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583384:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583393:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583405:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766476.583427:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076535424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583468:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766476.583470:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583477:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583484:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583489:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766476.583517:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076535552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583528:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766476.583533:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583549:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583561:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583569:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583579:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766476.583604:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076535680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583639:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766476.583642:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583649:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583660:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583667:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583682:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766476.583707:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076535808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766476.583758:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583775:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583782:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583795:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583810:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766476.583829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076535936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583866:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.583870:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766476.583876:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.583883:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.583901:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.583910:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.583926:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766476.583951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076536064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.583996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.584008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766476.584011:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.584060:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.584071:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.584081:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.584094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766476.584116:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076536192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766476.584143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766476.584147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766476.584149:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766476.584156:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2978 00000100:00080000:1.0:1775766476.584168:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766476.584176:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766476.584190:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766476.584225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076536320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766476.775150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766477.607131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766477.607205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766477.799242:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766478.631206:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766478.823113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766479.655133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766479.655190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766479.847186:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766480.679191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766480.871153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766481.703200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766481.703201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766481.703220:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766481.703229:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703258:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703278:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775766481.703291:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766481.703293:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766481.703367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076536448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.703411:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.703417:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766481.703420:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703425:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703430:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.703437:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.703452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766481.703475:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076536576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.703491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.703518:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766481.703522:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703528:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703534:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.703542:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.703554:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766481.703590:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076536704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.703613:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.703621:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766481.703625:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703633:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703643:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.703653:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.703664:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766481.703692:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076536832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.703708:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.703713:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766481.703715:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703721:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703727:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.703732:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.703751:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766481.703772:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076536960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.703814:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.703821:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766481.703829:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.703856:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.703863:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.703873:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.703881:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766481.703904:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076537088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.704100:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.704108:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766481.704112:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.704124:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.704134:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.704148:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.704166:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766481.704198:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076537216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.704230:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.704262:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766481.704281:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.704290:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.704298:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.704305:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.704314:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766481.704340:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076537344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766481.704356:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766481.704367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766481.704374:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766481.704391:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2983 00000100:00080000:1.0:1775766481.704397:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766481.704401:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766481.704407:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766481.704430:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076537472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766482.727147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766482.919236:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766483.751102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766483.751186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766483.943148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766484.775104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766484.967112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766485.799090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766485.799161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766485.991102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766486.823104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766486.823153:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766486.823166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823175:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823192:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766486.823273:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076537600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766486.823315:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823320:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766486.823323:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823328:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823333:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823338:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766486.823361:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076537600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766486.823362:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076537728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766486.823378:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823381:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766486.823382:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823385:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823389:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823391:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823394:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766486.823406:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076537856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766486.823424:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823427:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766486.823429:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823439:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2988 00000800:00000100:2.0:1775766486.823460:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766486.823461:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823466:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823472:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766486.823482:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076537984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766486.823489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766486.823493:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823496:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823499:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766486.823503:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350932300 x1862023076537600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766486.823504:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823511:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766486.823525:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076538112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766486.823530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076537600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766486.823538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823544:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766486.823545:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823548:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823550:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823553:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823555:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766486.823563:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076538240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766486.823572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.823580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766486.823585:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766486.823586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076537600:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766486.823591:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766486.823593:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766486.823602:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823605:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766486.823608:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766486.823621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766486.823624:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076538368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766486.823638:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766486.823643:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766486.823646:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766486.823647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766486.823655:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00080000:1.0:1775766486.823661:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766486.823663:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350932a00 x1862023076537728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766486.823669:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766486.823680:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766486.823683:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076537728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.823692:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.823696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766486.823700:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076538496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766486.823701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766486.823711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766486.823725:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775766486.823738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766486.823759:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766486.823762:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766486.823770:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350931c00 x1862023076537856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766486.823771:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2988 00000100:00100000:2.0:1775766486.823784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076537856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766486.823789:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766486.823805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766486.823828:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766486.823829:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.823839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537856:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766486.823841:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766486.823879:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076538624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766486.823911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766486.823941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.823951:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350930700 x1862023076537984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.823960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076537984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.823966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.823969:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.823971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076537984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766486.823975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076538112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766486.823988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.823994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350931880 x1862023076538112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.824001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076538112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.824007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.824009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.824011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076538112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766486.824014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766486.824024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.824030:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350932680 x1862023076538240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.824066:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076538240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.824073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.824075:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.824077:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766486.824092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766486.824108:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.824115:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350930a80 x1862023076538368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.824123:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076538368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.824129:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.824131:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.824133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766486.824136:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766486.824146:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.824151:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350930000 x1862023076538496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.824158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076538496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.824173:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.824176:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.824178:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766486.824201:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766486.824218:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766486.824235:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766486/real 1775766486] req@ffffa04350930e00 x1862023076538624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766486.824246:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076538624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766486.824253:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766486.824256:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766486.824258:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076538624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766487.015153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766487.847209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766487.847270:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766488.039128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766488.871095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766489.063128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766489.895160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766489.895260:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766490.087142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766490.919103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766491.111123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766491.943130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766491.943143:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775766491.943149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766491.943154:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943175:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943181:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766491.943217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076538752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766491.943225:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766491.943240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766491.943245:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766491.943252:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943256:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943261:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943273:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766491.943283:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076538752:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766491.943291:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076538880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766491.943318:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766491.943323:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766491.943326:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775766491.943346:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766491.943350:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076539008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766491.943357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943360:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766491.943361:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766491.943364:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943368:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943370:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943373:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766491.943380:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076539136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766491.943386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766491.943391:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766491.943393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350930e00 x1862023076538752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766491.943395:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943407:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943414:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943422:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766491.943432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076539264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766491.943438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076538752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766491.943453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766491.943459:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766491.943461:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766491.943467:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766491.943470:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00100000:2.0:1775766491.943476:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076538752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766491.943477:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943486:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943492:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766491.943504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076538880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766491.943508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076539392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766491.943522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766491.943529:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766491.943534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766491.943537:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943543:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766491.943553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350930000 x1862023076538880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766491.943562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766491.943576:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076539520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766491.943590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076538880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766491.943597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766491.943608:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766491.943614:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:2.0:1775766491.943615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766491.943618:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766491.943622:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766491.943625:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943630:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766491.943631:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076538880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766491.943642:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076539648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766491.943643:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539008:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766491.943657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766491.943660:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766491.943661:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766491.943665:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2993 00000100:00080000:1.0:1775766491.943668:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766491.943671:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766491.943675:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766491.943684:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766491.943696:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076539776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766491.943703:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350930a80 x1862023076539008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.943724:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076539008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.943747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.943751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.943755:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766491.943761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766491.943791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.943840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350932680 x1862023076539136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.943852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076539136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.943870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.943873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.943877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766491.943883:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766491.943908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.943918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350931880 x1862023076539264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.943931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076539264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.943941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.943944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.943952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076539264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766491.943969:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766491.943987:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.943997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350930700 x1862023076539392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.944019:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076539392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.944049:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.944052:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.944056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766491.944062:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766491.944079:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.944089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350931c00 x1862023076539520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.944098:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076539520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.944103:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.944105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.944107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766491.944111:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766491.944123:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.944136:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350932a00 x1862023076539648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.944144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076539648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.944149:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.944151:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.944153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766491.944156:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766491.944167:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766491.944172:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766491/real 1775766491] req@ffffa04350932300 x1862023076539776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766491.944179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076539776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766491.944183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766491.944185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766491.944187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076539776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766492.135137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766492.967264:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766493.159132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766493.991127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766493.991127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766494.183120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766495.015154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766495.207136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766496.039148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766496.039262:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766496.231135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766497.063157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063168:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775766497.063173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766497.063174:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063196:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063235:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063242:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766497.063285:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076539904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766497.063324:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063328:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063333:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063338:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063350:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766497.063370:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076539904:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766497.063379:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076540032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063390:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766497.063394:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063403:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063416:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063420:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766497.063436:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076540160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766497.063464:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063469:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063476:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063482:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766497.063487:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063493:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766497.063508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076540288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063519:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766497.063521:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063527:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063530:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063535:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766497.063541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350932300 x1862023076539904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766497.063543:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766497.063561:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076540416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766497.063575:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766497.063580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076539904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766497.063585:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063591:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063599:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063611:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775766497.063612:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.063621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766497.063623:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076540544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766497.063634:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076539904:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766497.063638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063649:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766497.063652:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063657:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063659:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063663:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063666:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766497.063673:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076540672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766497.063681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766497.063685:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766497.063691:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766497.063695:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063700:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766497.063710:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063714:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766497.063720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766497.063724:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076540800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766497.063732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766497.063735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766497.063743:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766497.063743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350932a00 x1862023076540032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.063782:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076540032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.063792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.063796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.063800:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766497.063805:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 2998 00000100:00080000:1.0:1775766497.063811:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766497.063816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766497.063821:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766497.063833:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766497.063845:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076540928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766497.063863:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.063880:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350931c00 x1862023076540160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.063910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076540160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.063920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.063924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.063927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766497.063934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766497.063958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.063970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350930700 x1862023076540288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.063982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076540288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.063991:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.063995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.063998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766497.064005:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766497.064024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.064065:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350931880 x1862023076540416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.064079:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076540416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.064103:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.064108:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.064111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076540416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766497.064118:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766497.064143:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.064156:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350932680 x1862023076540544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.064168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076540544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.064178:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.064182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.064185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766497.064206:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766497.064227:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.064238:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350930a80 x1862023076540672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.064265:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076540672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.064274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.064278:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.064288:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766497.064295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766497.064318:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.064330:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350930000 x1862023076540800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.064343:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076540800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.064352:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.064355:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.064359:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766497.064365:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766497.064386:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766497.064413:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766497/real 1775766497] req@ffffa04350930e00 x1862023076540928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766497.064426:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076540928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766497.064434:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766497.064438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766497.064441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076540928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766497.255231:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766498.087295:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766498.279148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766499.111163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766499.111218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766499.303137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766500.135196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766500.135257:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766500.327198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766501.159186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766501.159293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766501.351126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766502.183160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766502.183235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766502.183274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183284:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766502.183289:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183302:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183314:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183323:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766502.183368:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076541056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183409:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183414:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766502.183415:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183420:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183427:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183435:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183447:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766502.183470:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076541184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766502.183471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076541056:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766502.183503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183506:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766502.183522:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183529:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183533:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183538:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183543:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766502.183557:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076541312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766502.183571:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183573:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183576:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183579:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183584:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766502.183591:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076541440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766502.183604:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183607:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183610:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183613:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183617:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766502.183634:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076541568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766502.183646:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766502.183650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766502.183654:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183663:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183667:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183671:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766502.183692:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076541696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183718:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183721:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766502.183723:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183727:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00100000:1.0:1775766502.183733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766502.183736:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183742:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183748:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766502.183765:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076541824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183775:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766502.183779:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183793:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183798:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766502.183800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541440:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766502.183805:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766502.183817:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766502.183832:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076541952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766502.183843:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766502.183846:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766502.183848:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766502.183852:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3003 00000100:00080000:0.0:1775766502.183855:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766502.183860:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766502.183867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541568:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775766502.183869:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766502.183889:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076542080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766502.183925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766502.183954:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766502.183976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766502.184003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766502.375158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766503.207234:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766503.207321:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766503.399119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766504.231111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766504.231205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766504.423130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766505.255105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766505.255333:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775766505.255494:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775766505.255579:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775766505.255609:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255722:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349641c00 x1862023076541056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775766505.255733:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775766505.255771:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076541056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.255793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1775766505.255795:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:1.0:1775766505.255803:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.255809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076541056:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766505.255845:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255854:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349640700 x1862023076542080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775766505.255867:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076542080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775766505.255889:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:1.0:1775766505.255890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.255901:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000800:00000100:0.0:1775766505.255908:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766505.255932:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775766505.255935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349642680 x1862023076541568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.255961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076541568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.255972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.255976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.255986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766505.255993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349640a80 x1862023076541696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256019:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076541696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256079:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256084:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256093:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766505.256101:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349640e00 x1862023076541824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256116:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076541824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256124:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766505.256142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349643b80 x1862023076541952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076541952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256168:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076541952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766505.256180:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349640380 x1862023076541184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256203:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076541184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256213:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256217:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256220:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766505.256228:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349643100 x1862023076541312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256240:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076541312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256249:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256252:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256256:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766505.256264:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766502/real 1775766505] req@ffffa04349641500 x1862023076541440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766505.256275:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076541440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766505.256282:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766505.256285:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766505.256288:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076541440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766505.447104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766506.279191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766506.279219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766506.471101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766507.303146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766507.303158:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303168:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303176:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303184:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766507.303191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766507.303225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076542208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766507.303234:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766507.303258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766507.303267:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303274:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303279:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303284:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303295:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766507.303295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076542208:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766507.303316:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076542336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766507.303328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766507.303341:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303346:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303350:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303354:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303359:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766507.303368:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076542464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766507.303382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766507.303387:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766507.303388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766507.303392:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303396:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303401:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303404:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303408:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766507.303416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076542592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766507.303422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303425:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766507.303426:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303429:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303432:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303436:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766507.303437:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349642300 x1862023076542208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766507.303450:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766507.303461:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076542208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766507.303462:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076542720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766507.303483:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766507.303484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766507.303492:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766507.303492:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.303499:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076542208:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766507.303500:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303507:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303510:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303513:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766507.303521:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076542848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766507.303527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766507.303530:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303534:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303538:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766507.303539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766507.303547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303553:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766507.303568:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076542976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766507.303580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775766507.303586:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766507.303595:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766507.303601:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00100000:1.0:1775766507.303604:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349641880 x1862023076542336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766507.303608:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766507.303618:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303626:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766507.303626:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076542336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.303657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766507.303666:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076543104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766507.303666:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.303671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766507.303678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766507.303695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766507.303699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766507.303703:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766507.303708:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766507.303712:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3008 00000100:00080000:0.0:1775766507.303724:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766507.303728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349643800 x1862023076542464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766507.303746:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766507.303763:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766507.303773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076542464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766507.303781:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076543232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766507.303787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.303797:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.303804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766507.303813:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766507.303841:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.303869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349642a00 x1862023076542592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.303881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076542592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.303890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.303894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.303898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766507.303904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766507.303927:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.303937:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349641180 x1862023076542720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.303957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076542720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.303979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.303982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.303986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076542720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766507.304001:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766507.304061:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.304074:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349641500 x1862023076542848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.304087:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076542848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.304097:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.304101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.304104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766507.304132:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766507.304154:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.304165:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349643100 x1862023076542976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.304176:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076542976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.304184:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.304187:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.304198:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076542976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766507.304205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076543104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766507.304228:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.304240:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349640380 x1862023076543104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.304251:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076543104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.304259:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.304262:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.304265:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076543104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766507.304271:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076543232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766507.304290:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766507.304301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766507/real 1775766507] req@ffffa04349643b80 x1862023076543232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766507.304315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076543232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766507.304323:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766507.304326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766507.304329:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076543232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766507.495106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766508.327273:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766508.327324:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766508.519119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766509.351165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766509.351170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766509.543116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766510.375150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766510.375166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766510.567113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766511.399151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766511.399228:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766511.591140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766512.423108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766512.423122:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423128:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766512.423131:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423156:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00080000:0.0:1775766512.423164:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423171:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766512.423177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766512.423202:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076543360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766512.423225:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423228:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766512.423230:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423233:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00080000:0.0:1775766512.423236:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423239:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423246:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766512.423252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076543360:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766512.423257:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076543488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766512.423265:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423267:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766512.423269:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423272:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00080000:0.0:1775766512.423273:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423276:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423279:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766512.423287:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076543616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766512.423302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423305:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766512.423308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766512.423309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766512.423313:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00080000:0.0:1775766512.423317:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423320:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423324:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766512.423334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076543744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766512.423340:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766512.423344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766512.423347:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349640e00 x1862023076543360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766512.423350:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00080000:0.0:1775766512.423355:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423362:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423368:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766512.423373:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076543360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766512.423382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076543872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766512.423383:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423389:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423396:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076543360:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766512.423404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766512.423410:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423416:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00100000:1.0:1775766512.423418:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766512.423428:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423433:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766512.423438:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423440:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766512.423448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349640a80 x1862023076543488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766512.423455:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076544000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766512.423461:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076543488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766512.423470:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766512.423472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766512.423473:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766512.423477:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766512.423479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766512.423483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00100000:1.0:1775766512.423486:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766512.423488:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423494:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543616:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766512.423500:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766512.423514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766512.423518:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076544128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766512.423528:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766512.423530:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766512.423532:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00100000:1.0:1775766512.423537:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349642680 x1862023076543616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766512.423542:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423548:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076543616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766512.423561:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766512.423568:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423572:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766512.423575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076544256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766512.423578:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766512.423603:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543744:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766512.423619:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766512.423622:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766512.423625:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766512.423628:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766512.423632:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3014 00000100:00100000:1.0:1775766512.423635:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349640700 x1862023076543744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766512.423638:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766512.423645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766512.423651:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766512.423652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076543744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423660:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766512.423664:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076544384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766512.423665:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766512.423683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766512.423697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349641c00 x1862023076543872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766512.423712:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076543872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423723:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423725:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076543872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766512.423741:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766512.423754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423761:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349640e00 x1862023076544000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766512.423768:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076544000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766512.423780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766512.423789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349643b80 x1862023076544128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766512.423800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076544128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766512.423817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766512.423829:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423835:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349640380 x1862023076544256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766512.423842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076544256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766512.423853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766512.423862:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766512.423867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766512/real 1775766512] req@ffffa04349643100 x1862023076544384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766512.423874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076544384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766512.423879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766512.423881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766512.423882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076544384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766512.615115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766513.447111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766513.447184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766513.639127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766514.471148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766514.471210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766514.663113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766515.495126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766515.495225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766515.687109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766516.519164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766516.519386:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766516.711120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766517.543139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766517.543183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766517.543310:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076544512:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775766517.543376:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543415:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825ea00 x1862023076544512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543439:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076544512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543467:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543476:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543485:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076544512:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766517.543510:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766517.543549:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825e680 x1862023076544640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076544640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543616:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766517.543631:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766517.543659:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543678:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825c000 x1862023076544768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076544768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766517.543702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766517.543721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825ed80 x1862023076544896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076544896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543756:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076544896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766517.543766:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766517.543781:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543788:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825e300 x1862023076545024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076545024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766517.543815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766517.543828:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825f100 x1862023076545152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076545152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766517.543863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766517.543874:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825d500 x1862023076545280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076545280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766517.543903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766517.543915:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825dc00 x1862023076545408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076545408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543933:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543935:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543937:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766517.543940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766517.543948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766517.543953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766517/real 1775766517] req@ffffa0435825d880 x1862023076545536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766517.543967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076545536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766517.543973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766517.543975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766517.543977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076545536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766517.735089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766518.567098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766518.567136:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766518.759117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766519.591122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766519.591171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766519.783093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766520.615177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766520.615212:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766520.807100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766521.639123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766521.639218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766521.831115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766522.663166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766522.663187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766522.663225:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766522.663237:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663249:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663256:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766522.663303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076545664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766522.663339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766522.663346:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766522.663351:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663356:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663361:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663373:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766522.663375:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076545664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766522.663390:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076545792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766522.663402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766522.663408:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766522.663412:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663416:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663420:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766522.663437:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076545920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766522.663458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766522.663467:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766522.663467:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766522.663476:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663502:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663512:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663519:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766522.663534:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076546048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766522.663538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349641500 x1862023076545664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766522.663565:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663569:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766522.663571:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766522.663574:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076545664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766522.663579:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663588:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766522.663591:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766522.663593:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766522.663598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766522.663602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766522.663603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076545664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766522.663618:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076546176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766522.663629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766522.663633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766522.663635:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766522.663639:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766522.663645:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3024 00000800:00000100:1.0:1775766522.663655:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766522.663658:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663666:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766522.663668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349641180 x1862023076545792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766522.663675:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766522.663680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076545792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.663689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766522.663694:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076546304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766522.663695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.663702:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766522.663718:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766522.663719:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766522.663722:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766522.663724:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766522.663731:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663736:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766522.663739:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663745:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766522.663761:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076546432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766522.663762:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766522.663774:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663778:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766522.663781:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766522.663781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349642a00 x1862023076545920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766522.663790:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663799:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766522.663804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076545920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766522.663808:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663817:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766522.663820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.663826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.663831:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076545920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766522.663834:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076546560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766522.663845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546048:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766522.663862:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766522.663867:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766522.663869:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766522.663874:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766522.663879:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3024 00000100:00080000:0.0:1775766522.663885:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766522.663886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349643800 x1862023076546048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766522.663894:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766522.663904:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766522.663905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076546048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.663917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.663923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766522.663924:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076546688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766522.663930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766522.663945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766522.663968:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766522.663977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349641880 x1862023076546176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766522.663996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076546176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.664003:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.664005:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.664007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766522.664023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766522.664058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766522.664066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349642300 x1862023076546304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766522.664074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076546304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.664080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.664083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.664085:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766522.664089:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766522.664099:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766522.664104:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349641500 x1862023076546432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766522.664114:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076546432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.664138:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.664142:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.664146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766522.664153:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766522.664174:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766522.664182:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349643100 x1862023076546560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766522.664190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076546560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.664196:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.664199:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.664201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766522.664204:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766522.664215:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766522.664220:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766522/real 1775766522] req@ffffa04349640380 x1862023076546688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766522.664228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076546688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766522.664233:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766522.664235:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766522.664241:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076546688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766522.855161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766523.687116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766523.687169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766523.879154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766524.711171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766524.903201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766525.735179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766525.735246:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766525.927122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766526.759170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766526.951189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766527.783105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766527.783109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766527.783117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766527.783122:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783132:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783143:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783151:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766527.783191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076546816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766527.783216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766527.783235:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783238:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783243:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766527.783246:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076546816:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766527.783249:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766527.783259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766527.783272:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076546944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766527.783280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766527.783284:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783287:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783296:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783299:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766527.783304:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766527.783305:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766527.783315:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076547072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766527.783322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766527.783326:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783328:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783331:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783334:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783338:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349643b80 x1862023076546816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766527.783345:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766527.783358:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076547200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766527.783360:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076546816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766527.783377:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766527.783381:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783385:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783387:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783390:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766527.783390:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766527.783396:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766527.783400:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766527.783404:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076547328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766527.783409:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076546816:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766527.783415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783426:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766527.783441:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783444:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783448:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783454:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783455:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546944:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766527.783458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766527.783473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076547456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766527.783483:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766527.783484:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775766527.783496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349640e00 x1862023076546944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766527.783498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783509:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00100000:1.0:1775766527.783513:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076546944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766527.783515:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766527.783524:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766527.783525:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766527.783531:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766527.783538:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783541:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076546944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766527.783546:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076547584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766527.783550:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766527.783563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766527.783567:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783571:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:0.0:1775766527.783575:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766527.783579:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766527.783582:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766527.783587:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766527.783592:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349641c00 x1862023076547072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076547072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766527.783606:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076547712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766527.783620:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766527.783624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766527.783627:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766527.783628:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766527.783631:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3029 00000100:00080000:1.0:1775766527.783635:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766527.783636:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766527.783639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766527.783645:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547200:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775766527.783652:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766527.783666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783675:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349640700 x1862023076547200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766527.783678:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076547840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766527.783694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076547200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766527.783715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766527.783735:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349642680 x1862023076547328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783749:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076547328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076547328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766527.783774:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766527.783784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349640a80 x1862023076547456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076547456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783805:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783806:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766527.783810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766527.783818:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349643b80 x1862023076547584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076547584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766527.783853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766527.783863:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349640380 x1862023076547712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076547712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766527.783907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766527.783927:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766527.783938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766527/real 1775766527] req@ffffa04349643100 x1862023076547840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766527.783950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076547840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766527.783966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766527.783968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766527.783970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076547840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766527.975112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766528.807137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766528.999135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766529.831122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766530.023115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766530.855121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766531.047126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766531.879124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766532.071111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766532.903148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766532.903158:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903167:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766532.903173:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903185:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903201:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903212:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766532.903264:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076547968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903304:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766532.903306:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903311:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903322:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903333:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766532.903342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076547968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766532.903348:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076548096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766532.903396:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903401:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903424:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903430:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766532.903438:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766532.903454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076548224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903466:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766532.903468:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903471:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903477:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903481:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766532.903489:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076548352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903495:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903498:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766532.903500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766532.903504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349641500 x1862023076547968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766532.903505:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903509:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903513:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903526:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766532.903535:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076548480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766532.903540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076547968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766532.903547:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766532.903558:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903565:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:1.0:1775766532.903568:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766532.903569:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903575:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766532.903577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766532.903580:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766532.903583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076547968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766532.903591:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076548608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903597:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766532.903601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903604:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903606:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903609:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766532.903614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766532.903622:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076548736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766532.903630:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903632:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775766532.903642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766532.903643:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903649:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903653:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903657:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766532.903660:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349642300 x1862023076548096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766532.903663:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766532.903680:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076548864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766532.903680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076548096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766532.903687:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766532.903694:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766532.903696:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766532.903700:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3034 00000100:00080000:0.0:1775766532.903702:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766532.903704:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766532.903709:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766532.903717:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766532.903722:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076548992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766532.903725:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.903733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766532.903744:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766532.903767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.903782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349641880 x1862023076548224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.903796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076548224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.903805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.903809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.903813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766532.903819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766532.903841:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.903852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349643800 x1862023076548352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.903866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076548352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.903875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.903879:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.903889:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766532.903894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766532.903912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.903919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349642a00 x1862023076548480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.903927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076548480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.903934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.903936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.903938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076548480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766532.903942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766532.903953:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.903959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349641180 x1862023076548608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.903966:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076548608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.903971:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.903973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.903975:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766532.903990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766532.904001:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.904013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349641500 x1862023076548736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.904021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076548736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.904053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.904055:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.904058:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766532.904063:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766532.904080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.904087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349643100 x1862023076548864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.904095:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076548864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.904100:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.904102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.904104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766532.904107:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766532.904117:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766532.904122:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766532/real 1775766532] req@ffffa04349640380 x1862023076548992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766532.904129:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076548992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766532.904140:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766532.904142:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766532.904145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076548992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766533.095115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766533.927153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766534.119136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766534.951106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766535.143146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766535.975185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766536.167198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766536.999170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766536.999215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766537.191164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766538.023166:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766538.023170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766538.023187:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766538.023193:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023210:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023255:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023265:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766538.023347:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076549120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766538.023377:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.023380:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766538.023382:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023387:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023393:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023403:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766538.023416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766538.023431:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076549248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766538.023444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076549120:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766538.023446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.023460:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766538.023462:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023488:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766538.023496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766538.023509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076549376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766538.023517:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.023529:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766538.023530:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023552:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3039 00000800:00000100:1.0:1775766538.023561:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766538.023565:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023573:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766538.023584:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766538.023601:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076549504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766538.023614:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766538.023615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349643b80 x1862023076549120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766538.023621:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766538.023626:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023641:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023667:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766538.023673:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076549120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766538.023681:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766538.023698:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766538.023698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.023705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.023713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076549120:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766538.023745:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076549632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766538.023754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766538.023772:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.023777:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766538.023780:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023787:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023795:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766538.023802:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766538.023809:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766538.023819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349640a80 x1862023076549248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766538.023833:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766538.023840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076549248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766538.023859:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076549760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766538.023868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.023881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766538.023885:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766538.023886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766538.023896:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766538.023900:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766538.023907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766538.023913:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00080000:0.0:1775766538.023925:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023932:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766538.023940:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766538.023956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766538.023960:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076549888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766538.023969:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.023973:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766538.023975:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766538.023981:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00100000:1.0:1775766538.023986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349642680 x1862023076549376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766538.023989:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.023998:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076549376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766538.024028:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766538.024041:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076550016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766538.024050:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766538.024055:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766538.024062:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766538.024063:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766538.024087:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766538.024096:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3039 00000100:00100000:1.0:1775766538.024099:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766538.024106:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766538.024114:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024115:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766538.024151:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024163:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349640700 x1862023076549504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766538.024164:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766538.024186:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076549504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766538.024191:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076550144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766538.024244:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024252:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024263:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766538.024289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766538.024308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024317:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349641c00 x1862023076549632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766538.024358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076549632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766538.024369:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024375:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024379:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076549632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766538.024386:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076549760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766538.024413:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024430:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349640e00 x1862023076549760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766538.024440:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076549760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766538.024447:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024449:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024452:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076549760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766538.024457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076549888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766538.024471:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024499:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349643b80 x1862023076549888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766538.024510:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076549888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766538.024520:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024524:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024528:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076549888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766538.024536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766538.024573:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024587:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349640380 x1862023076550016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766538.024602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076550016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766538.024613:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024621:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766538.024629:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766538.024653:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766538.024677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766538/real 1775766538] req@ffffa04349643100 x1862023076550144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766538.024692:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076550144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766538.024711:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766538.024716:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766538.024720:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766538.215148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766539.047128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766539.047371:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766539.239131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766540.071111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766540.263124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766541.095164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766541.095355:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766541.287121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766542.119212:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766542.311234:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775766543.143177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766543.143193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766543.143212:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143228:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766543.143235:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143263:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143297:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143306:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766543.143367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076550272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766543.143403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143407:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766543.143409:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143414:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143418:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143424:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.143437:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766543.143452:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076550272:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766543.143454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076550400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766543.143504:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143508:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766543.143511:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143526:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143532:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143536:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.143542:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766543.143559:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076550528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766543.143575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766543.143596:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766543.143603:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143610:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143616:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143631:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.143639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766543.143649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350930e00 x1862023076550272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766543.143654:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076550656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766543.143702:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143713:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766543.143721:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143726:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00100000:2.0:1775766543.143735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076550272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766543.143740:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143755:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766543.143756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766543.143763:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766543.143765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.143777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076550272:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766543.143784:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076550784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766543.143804:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143810:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766543.143812:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766543.143817:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143820:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143825:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.143834:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766543.143837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766543.143851:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076550912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766543.143866:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.143874:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766543.143879:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766543.143879:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766543.143892:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.143898:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.143902:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.143909:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766543.143917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350930000 x1862023076550400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766543.143921:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076551040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766543.143982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076550400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766543.143988:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.144003:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766543.144010:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766543.144014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766543.144022:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:2.0:1775766543.144028:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144032:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766543.144043:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766543.144066:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766543.144071:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.144080:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766543.144103:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076551168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766543.144117:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766543.144119:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766543.144124:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766543.144127:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766543.144132:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350930a80 x1862023076550528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766543.144136:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3044 00000100:00080000:1.0:1775766543.144150:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766543.144156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076550528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766543.144161:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766543.144173:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775766543.144176:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766543.144199:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076551296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766543.144209:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766543.144257:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144286:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350932680 x1862023076550656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076550656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144322:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144329:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766543.144337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766543.144359:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144372:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350931880 x1862023076550784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144386:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076550784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144400:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144403:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076550784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766543.144424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766543.144446:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144458:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350930700 x1862023076550912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144471:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076550912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144497:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144501:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144505:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076550912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766543.144530:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766543.144563:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144578:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350931c00 x1862023076551040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076551040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144597:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144600:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766543.144609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766543.144644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350932a00 x1862023076551168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144659:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076551168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766543.144698:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766543.144723:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766543.144741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766543/real 1775766543] req@ffffa04350932300 x1862023076551296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766543.144752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076551296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766543.144758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766543.144760:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766543.144763:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076551296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766543.335095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766544.167234:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766544.359165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766545.191205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766545.383199:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766546.215194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766546.407135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766547.239204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766547.431184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766548.263127:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263137:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766548.263143:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263155:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00100000:1.0:1775766548.263158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766548.263169:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263179:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766548.263253:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076551424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263290:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263293:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766548.263295:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263299:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263303:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263307:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263315:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766548.263326:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076551552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263334:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263337:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766548.263338:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263341:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263344:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263347:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263350:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766548.263353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076551424:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766548.263368:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076551680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263377:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263381:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766548.263384:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263410:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263419:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263426:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766548.263429:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263439:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766548.263457:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076551808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263467:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766548.263469:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349641500 x1862023076551424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766548.263474:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766548.263478:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263488:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263493:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766548.263497:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076551424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766548.263510:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766548.263511:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766548.263519:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766548.263519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.263525:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076551424:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766548.263533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766548.263536:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076551936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263546:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263549:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766548.263551:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263557:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3049 00000800:00000100:2.0:1775766548.263559:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766548.263561:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263564:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263569:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766548.263581:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076552064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766548.263585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349641180 x1862023076551552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.263616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076551552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766548.263617:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263627:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766548.263631:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766548.263639:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766548.263640:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263648:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766548.263651:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766548.263655:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263662:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766548.263663:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766548.263676:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076552192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263683:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263686:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766548.263687:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263691:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3049 00000100:00080000:0.0:1775766548.263694:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766548.263698:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766548.263699:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766548.263706:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766548.263717:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076552320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766548.263726:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766548.263729:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766548.263731:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766548.263737:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3049 00000800:00000100:2.0:1775766548.263738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766548.263747:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766548.263752:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766548.263756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349642a00 x1862023076551680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766548.263758:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766548.263776:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076552448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766548.263776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076551680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.263808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.263815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.263819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766548.263827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766548.263843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.263851:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349643800 x1862023076551808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.263862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076551808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.263882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.263886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.263890:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766548.263897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766548.263923:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.263936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349641880 x1862023076551936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.263946:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076551936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.263954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.263956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.263958:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076551936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766548.263964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766548.263979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.263987:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349642300 x1862023076552064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.263995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076552064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.264001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.264003:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.264011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766548.264016:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766548.264058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.264068:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349643100 x1862023076552192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.264078:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076552192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.264084:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.264086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.264089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766548.264106:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766548.264120:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.264127:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349640380 x1862023076552320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.264135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076552320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.264141:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.264143:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.264145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766548.264149:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766548.264160:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766548.264174:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766548/real 1775766548] req@ffffa04349643b80 x1862023076552448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766548.264183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076552448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766548.264189:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766548.264192:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766548.264194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076552448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766548.455152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766549.287157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766549.479136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766550.311184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766550.503162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766551.335262:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766551.527145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766552.359145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766552.359540:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766552.551236:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766553.383240:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766553.383249:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766553.383256:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766553.383265:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383280:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383294:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383301:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766553.383351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076552576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383389:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766553.383391:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383397:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383403:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383409:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766553.383435:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076552576:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766553.383437:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766553.383460:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076552704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383474:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766553.383497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383504:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383511:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383515:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766553.383522:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775766553.383535:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766553.383540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076552832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383554:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383560:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766553.383562:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383567:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383573:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383578:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766553.383591:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349640e00 x1862023076552576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766553.383597:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766553.383620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076552960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383632:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766553.383635:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076552576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766553.383642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766553.383657:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383684:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:2.0:1775766553.383686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766553.383689:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383704:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766553.383704:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766553.383719:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766553.383727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076552576:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766553.383735:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076553088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383743:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383746:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766553.383748:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383751:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383757:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383762:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766553.383767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552704:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766553.383770:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766553.383788:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076553216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766553.383797:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383800:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766553.383801:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383811:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3055 00000800:00000100:2.0:1775766553.383812:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766553.383819:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383825:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766553.383830:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766553.383840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349641c00 x1862023076552704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766553.383852:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076553344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766553.383873:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076552704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766553.383875:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383893:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766553.383900:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383905:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383908:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383911:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766553.383914:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766553.383931:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076553472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766553.383934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.383953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766553.383954:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766553.383959:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766553.383960:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766553.383964:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3055 00000100:00080000:0.0:1775766553.383965:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766553.383968:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766553.383971:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766553.383974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766553.383986:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076553600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766553.383990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766553.384069:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349640700 x1862023076552832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384105:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076552832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384113:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766553.384126:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766553.384143:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349642680 x1862023076552960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384182:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076552960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384199:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384202:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076552960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766553.384219:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766553.384240:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384250:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349640a80 x1862023076553088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076553088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384268:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384271:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384274:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766553.384295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766553.384309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384317:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349640e00 x1862023076553216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384327:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076553216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384334:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384337:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384340:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766553.384345:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766553.384358:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384374:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349643b80 x1862023076553344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384385:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076553344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384393:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384395:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384398:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766553.384403:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766553.384418:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384425:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349640380 x1862023076553472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384435:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076553472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384442:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384444:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384447:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766553.384461:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766553.384474:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766553.384481:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766553/real 1775766553] req@ffffa04349643100 x1862023076553600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766553.384491:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076553600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766553.384503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766553.384506:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766553.384509:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076553600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766553.575243:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766554.407239:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766554.407289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766554.599252:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766555.431097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766555.623124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766556.455235:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766556.455380:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766556.647116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766557.479161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766557.671143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766558.503103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766558.503143:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503149:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766558.503164:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503172:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503181:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775766558.503184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766558.503188:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766558.503221:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076553728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503245:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766558.503247:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503249:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503252:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503255:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503262:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766558.503271:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076553856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503280:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503284:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775766558.503284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076553728:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766558.503287:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503294:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503297:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503314:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503318:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766558.503327:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076553984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503334:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503336:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766558.503338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503340:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503342:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503347:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766558.503351:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503353:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766558.503361:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076554112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503367:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766558.503371:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503374:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503376:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503378:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766558.503388:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076554240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503394:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766558.503399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349642300 x1862023076553728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766558.503419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766558.503424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503429:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00100000:2.0:1775766558.503433:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076553728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766558.503435:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503445:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775766558.503446:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503454:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766558.503455:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766558.503458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076553728:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766558.503473:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076554368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766558.503480:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766558.503489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766558.503494:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503498:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:0.0:1775766558.503503:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766558.503507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766558.503510:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503515:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766558.503521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349641880 x1862023076553856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766558.503528:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076554496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766558.503537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076553856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766558.503538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503546:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766558.503549:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503565:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3060 00000100:00080000:2.0:1775766558.503566:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766558.503567:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766558.503572:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766558.503573:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503585:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553856:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766558.503587:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766558.503593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766558.503599:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076554624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766558.503608:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766558.503610:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766558.503611:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766558.503616:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3060 00000800:00000100:2.0:1775766558.503616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766558.503633:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766558.503637:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766558.503641:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766558.503643:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349643800 x1862023076553984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766558.503651:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076554752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766558.503657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076553984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503670:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503673:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503676:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076553984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766558.503682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076554112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766558.503694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503700:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349642a00 x1862023076554112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076554112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503716:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076554112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766558.503720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076554240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766558.503729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349641180 x1862023076554240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076554240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076554240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766558.503774:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766558.503785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349641500 x1862023076554368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503798:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076554368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503805:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503807:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766558.503810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766558.503819:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503824:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349642300 x1862023076554496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503831:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076554496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503840:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503843:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766558.503849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766558.503861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349643100 x1862023076554624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076554624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503883:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766558.503886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766558.503896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766558.503901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766558/real 1775766558] req@ffffa04349640380 x1862023076554752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766558.503908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076554752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766558.503912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766558.503914:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766558.503921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076554752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766558.695090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766559.527274:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766559.719152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766560.551213:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766560.551272:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766560.743175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766561.575156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766561.767112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766562.599151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766562.599249:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766562.791249:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766563.623125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766563.623144:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623150:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766563.623154:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623163:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623172:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766563.623178:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766563.623225:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076554880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766563.623249:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623252:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766563.623253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623256:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623259:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766563.623262:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623269:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766563.623278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076555008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766563.623296:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623299:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766563.623301:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766563.623307:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076554880:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766563.623308:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623314:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766563.623319:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766563.623337:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076555136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766563.623347:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623350:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766563.623352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623363:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623368:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766563.623372:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766563.623376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766563.623397:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076555264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766563.623406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766563.623410:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349643b80 x1862023076554880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766563.623412:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766563.623416:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623426:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623430:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766563.623434:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076554880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766563.623438:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623449:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766563.623450:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623457:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623461:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076554880:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766563.623466:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076555392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766563.623468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555008:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766563.623483:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623486:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766563.623488:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766563.623493:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766563.623498:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623503:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766563.623504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349640e00 x1862023076555008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766563.623511:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623520:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766563.623522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076555008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623531:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766563.623535:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076555520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766563.623537:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623540:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555008:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766563.623551:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766563.623555:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623559:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623563:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766563.623565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555136:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766563.623569:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766563.623586:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076555648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766563.623589:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766563.623595:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766563.623600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766563.623601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766563.623605:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00080000:0.0:1775766563.623608:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766563.623612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623612:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349640a80 x1862023076555136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766563.623618:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766563.623630:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076555776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766563.623631:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076555136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766563.623657:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766563.623658:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766563.623662:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766563.623665:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766563.623665:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555136:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766563.623673:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3065 00000100:00100000:2.0:1775766563.623679:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555264:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766563.623694:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766563.623700:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766563.623704:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766563.623709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766563.623717:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076555904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766563.623720:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349642680 x1862023076555264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076555264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623754:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766563.623759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766563.623769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349640700 x1862023076555392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076555392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076555392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766563.623799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766563.623811:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349641c00 x1862023076555520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076555520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623829:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623831:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623833:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766563.623845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766563.623854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623860:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349640e00 x1862023076555648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076555648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623880:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766563.623884:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766563.623896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349643b80 x1862023076555776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076555776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766563.623921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766563.623929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766563.623934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766563/real 1775766563] req@ffffa04349640380 x1862023076555904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766563.623942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076555904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766563.623946:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766563.623948:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766563.623950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076555904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766563.815137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766564.647231:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766564.647317:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766564.839132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766565.671132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766565.671176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766565.863112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766566.695147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766566.695199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766566.887148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766567.719119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766567.719157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766567.911170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766568.743220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766568.743226:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766568.743246:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743261:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3070 00010000:00010000:2.0:1775766568.743265:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766568.743292:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743303:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766568.743357:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076556032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743390:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743394:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766568.743396:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743400:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743403:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743407:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766568.743439:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076556032:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766568.743446:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766568.743478:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076556160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766568.743496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743501:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743509:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743514:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743519:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766568.743529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076556288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743545:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766568.743547:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743551:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743554:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743557:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743562:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766568.743575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076556416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743610:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743616:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775766568.743629:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766568.743631:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743653:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743663:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743672:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743681:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766568.743698:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076556544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766568.743712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556288:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766568.743716:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743728:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766568.743730:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743735:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743740:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743746:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743753:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766568.743770:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076556672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766568.743782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556416:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766568.743791:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743796:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766568.743798:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743804:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743811:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743817:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743822:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766568.743835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766568.743854:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076556800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743872:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743877:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766568.743879:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743892:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00080000:0.0:1775766568.743897:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766568.743900:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766568.743906:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.743916:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766568.743933:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076556928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766568.743947:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766568.743955:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766568.743958:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766568.743965:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3070 00000100:00100000:2.0:1775766568.743974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766568.743975:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766568.743993:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766568.744101:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766568.744110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766568.744132:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076557056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766568.744175:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557056:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775766569.767194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766569.767267:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766569.959189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766570.791232:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766570.791305:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766570.983158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766571.815152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766571.815411:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775766571.815427:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775766571.815478:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775766571.815495:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815547:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815561:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815574:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815586:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766571.815588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349643b80 x1862023076557056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775766571.815597:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815611:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766571.815627:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076557056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775766571.815633:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775766571.815648:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:2.0:1775766571.815649:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815663:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766571.815673:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349641880 x1862023076556800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815713:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076556800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815723:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815726:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766571.815736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349640380 x1862023076556928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815749:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076556928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815763:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766571.815770:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349642a00 x1862023076556544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076556544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815788:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815791:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815794:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766571.815800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349643800 x1862023076556672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076556672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815837:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076556672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766571.815844:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349642300 x1862023076556160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076556160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815865:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766571.815884:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349641500 x1862023076556288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076556288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766571.815917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349641180 x1862023076556416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076556416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815945:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815948:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.815952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076556416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766571.815958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766568/real 1775766571] req@ffffa04349643100 x1862023076556032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766571.815988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076556032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766571.815995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766571.815999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766571.816001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076556032:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766572.007121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766572.839286:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766572.839337:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766573.031202:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766573.863156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766573.863328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076557184:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766573.863407:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863473:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825ce00 x1862023076557184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863492:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076557184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.863508:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863512:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076557184:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766573.863536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766573.863560:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825d880 x1862023076557312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863605:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076557312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.863619:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863623:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766573.863648:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766573.863671:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863683:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825dc00 x1862023076557440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863696:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076557440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863706:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.863710:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863714:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766573.863721:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766573.863741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825d500 x1862023076557568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076557568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.863784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766573.863806:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766573.863831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825f100 x1862023076557696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863857:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076557696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:2.0:1775766573.863869:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766573.863870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076557696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766573.863880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766573.863901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.863914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825e300 x1862023076557824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.863927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076557824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.863936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.863939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.863981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766573.864000:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766573.864070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.864086:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825ed80 x1862023076557952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.864101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076557952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.864111:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.864115:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.864119:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076557952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766573.864126:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766573.864149:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.864163:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825c000 x1862023076558080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.864175:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076558080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.864184:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.864188:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.864191:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766573.864197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766573.864219:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766573.864242:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766573/real 1775766573] req@ffffa0435825e680 x1862023076558208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766573.864254:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076558208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766573.864262:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766573.864266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766573.864269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766574.055118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766574.887289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766575.079110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766575.911165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766575.911206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766576.103098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766576.935141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766576.935375:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766577.127160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766577.959140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766577.959336:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766578.151101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766578.983180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766578.983366:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076558336:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766578.983452:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.983500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825ea00 x1862023076558336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.983542:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076558336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.983561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.983570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.983576:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076558336:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766578.983617:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766578.983681:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.983707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825e680 x1862023076558464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.983749:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076558464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.983771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.983780:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.983785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766578.983838:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766578.983866:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.983879:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825c000 x1862023076558592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.983889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076558592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.983912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.983916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.983919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766578.983926:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558720:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775766578.983933:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1775766578.983947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.983955:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825ed80 x1862023076558720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.983970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076558720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.983976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.983978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.983980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766578.983984:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766578.983996:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.984003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825e300 x1862023076558848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.984011:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076558848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.984020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.984073:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.984077:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076558848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766578.984082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766578.984100:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.984108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825f100 x1862023076558976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.984116:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076558976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.984131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.984134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.984136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076558976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766578.984140:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766578.984154:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.984160:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825d500 x1862023076559104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.984168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076559104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.984173:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.984174:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.984176:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766578.984180:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766578.984190:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.984196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825dc00 x1862023076559232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.984203:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076559232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.984208:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.984209:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.984216:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766578.984222:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766578.984243:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766578.984255:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766578/real 1775766578] req@ffffa0435825d880 x1862023076559360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766578.984267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076559360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766578.984276:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766578.984280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766578.984282:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076559360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766579.175171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766580.007105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766580.007185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766580.199161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766581.031145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766581.031285:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766581.223109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766582.055105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766582.055228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766582.247127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766583.079142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766583.079219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766583.271178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766584.103104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766584.103293:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076559488:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766584.103352:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103392:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825ce00 x1862023076559488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103413:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076559488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103443:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103468:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076559488:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766584.103478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766584.103513:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103534:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825d880 x1862023076559616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103564:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076559616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103592:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766584.103622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766584.103644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825dc00 x1862023076559744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076559744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103680:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103685:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766584.103705:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766584.103728:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103737:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825d500 x1862023076559872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103747:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076559872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103761:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076559872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766584.103768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766584.103789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825f100 x1862023076560000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1775766584.103803:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766584.103812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076560000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766584.103835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766584.103873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825e300 x1862023076560128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076560128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766584.103910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766584.103923:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103929:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825ed80 x1862023076560256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076560256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.103942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.103943:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.103945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766584.103959:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766584.103970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.103976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825c000 x1862023076560384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.103983:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076560384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.104007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.104009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.104012:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766584.104016:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766584.104057:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766584.104066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766584/real 1775766584] req@ffffa0435825e680 x1862023076560512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766584.104074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076560512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766584.104079:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766584.104082:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766584.104084:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076560512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766584.295130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766585.127105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766585.127167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766585.319118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766586.151145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766586.151246:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766586.343127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766587.175106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766587.175178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766587.367130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766588.199141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766588.199142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766588.391101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766589.223133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766589.223277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076560640:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766589.223367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223426:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825ea00 x1862023076560640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076560640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223500:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223512:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223521:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076560640:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766589.223552:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766589.223581:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825e680 x1862023076560768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223604:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076560768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223613:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766589.223627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766589.223644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00010000:00010000:2.0:1775766589.223653:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766589.223659:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825c000 x1862023076560896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076560896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223696:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223700:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223703:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076560896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766589.223710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766589.223732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825ed80 x1862023076561024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076561024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223765:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766589.223778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766589.223832:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223844:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825e300 x1862023076561152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223857:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076561152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766589.223895:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766589.223918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223929:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825f100 x1862023076561280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.223941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076561280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.223951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.223953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.223956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766589.223961:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766589.223981:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.223991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825d500 x1862023076561408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.224003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076561408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.224011:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.224014:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.224047:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766589.224055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766589.224080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.224093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825dc00 x1862023076561536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.224105:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076561536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.224114:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.224117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.224121:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766589.224148:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766589.224168:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766589.224179:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766589/real 1775766589] req@ffffa0435825d880 x1862023076561664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766589.224191:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076561664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766589.224200:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766589.224203:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766589.224206:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076561664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766589.415102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766590.247143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766590.247238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766590.439132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766591.271134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766591.271145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766591.463100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766592.295109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766592.295158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766592.487090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766593.319111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766593.319176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766593.511110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766594.343138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766594.343157:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343165:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766594.343171:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775766594.343187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766594.343193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343206:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343214:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766594.343273:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076561792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766594.343302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343306:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766594.343308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343312:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343315:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343319:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766594.343342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076561920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766594.343350:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076561792:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766594.343368:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343390:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766594.343392:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343398:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343403:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343406:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343414:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766594.343424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076562048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766594.343437:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766594.343443:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343447:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766594.343448:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343452:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343456:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343460:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766594.343473:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076562176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766594.343480:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343484:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766594.343486:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343492:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343496:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343502:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766594.343503:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7e300 x1862023076561792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766594.343521:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766594.343538:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076561792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766594.343539:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076562304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766594.343614:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766594.343614:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343627:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766594.343631:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766594.343632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.343651:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076561792:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766594.343651:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343660:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343666:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343671:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766594.343683:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076562432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766594.343686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766594.343711:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343714:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766594.343716:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343721:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343723:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343729:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766594.343730:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343751:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766594.343776:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076562560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766594.343785:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343788:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766594.343794:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766594.343800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7f800 x1862023076561920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766594.343800:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343808:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343816:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343828:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766594.343848:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076562688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766594.343858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076561920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766594.343862:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766594.343880:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766594.343881:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766594.343889:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3096 00000100:00080000:2.0:1775766594.343900:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766594.343907:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766594.343914:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766594.343917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766594.343924:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076562816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766594.343929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.343952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076561920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766594.343960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766594.343983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.343992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7f100 x1862023076562048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076562048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344010:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344012:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766594.344017:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766594.344029:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344035:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7c000 x1862023076562176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344043:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076562176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344048:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344050:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344053:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766594.344077:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766594.344091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344107:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7dc00 x1862023076562304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344121:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076562304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344130:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344137:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076562304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766594.344162:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766594.344182:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344191:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7f480 x1862023076562432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076562432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344209:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344211:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766594.344215:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766594.344226:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344232:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7e680 x1862023076562560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344274:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076562560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344288:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344291:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766594.344295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766594.344329:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7c380 x1862023076562688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344347:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076562688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344353:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344355:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344357:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766594.344362:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766594.344382:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766594.344410:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766594/real 1775766594] req@ffffa0434aa7e300 x1862023076562816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766594.344422:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076562816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766594.344431:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766594.344434:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766594.344446:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076562816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766594.535127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766595.367103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766595.559128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766596.391219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766596.391429:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766596.583189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766597.415181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766597.607174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766598.439156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766598.439254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766598.631160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766599.463160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766599.463184:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766599.463194:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766599.463199:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463209:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463217:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463242:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766599.463278:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076562944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766599.463306:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766599.463309:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766599.463311:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463314:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463318:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463324:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463333:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766599.463350:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076563072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766599.463351:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076562944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766599.463378:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766599.463382:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766599.463384:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463391:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463396:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463401:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463407:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766599.463422:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076563200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766599.463430:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766599.463433:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766599.463440:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766599.463445:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766599.463450:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463455:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463459:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463467:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766599.463481:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076563328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766599.463493:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766599.463494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350932300 x1862023076562944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766599.463511:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766599.463519:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463524:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463528:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766599.463531:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076562944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766599.463536:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463552:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775766599.463557:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.463562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766599.463565:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076563456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766599.463569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076562944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766599.463590:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766599.463593:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766599.463594:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463599:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463601:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463604:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463610:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766599.463612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766599.463653:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076563584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766599.463654:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766599.463671:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766599.463674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350932a00 x1862023076563072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766599.463696:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766599.463699:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463728:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463733:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766599.463737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076563072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766599.463740:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463744:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766599.463754:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076563712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766599.463776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766599.463790:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766599.463794:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766599.463806:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766599.463810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766599.463811:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766599.463828:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00100000:0.0:1775766599.463831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766599.463831:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463840:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766599.463845:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766599.463860:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076563840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766599.463870:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766599.463872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766599.463879:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766599.463881:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766599.463894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350931c00 x1862023076563200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766599.463901:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3101 00000100:00080000:1.0:1775766599.463911:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766599.463916:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766599.463922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076563200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766599.463924:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766599.463932:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.463935:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.463942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766599.463953:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766599.463953:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076563968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766599.463988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464000:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350930700 x1862023076563328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076563328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766599.464052:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766599.464069:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350931880 x1862023076563456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464085:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076563456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464091:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464093:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464095:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076563456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766599.464113:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766599.464124:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350932680 x1862023076563584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076563584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464144:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464146:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464147:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766599.464151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766599.464161:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464175:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350930a80 x1862023076563712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076563712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464189:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464191:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464193:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766599.464197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766599.464208:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464214:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350930000 x1862023076563840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464221:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076563840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464226:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464228:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464229:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766599.464233:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766599.464243:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766599.464248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766599/real 1775766599] req@ffffa04350930e00 x1862023076563968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766599.464255:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076563968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766599.464265:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766599.464267:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766599.464269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076563968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766599.655144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766600.487202:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766600.487324:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766600.679147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766601.511179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766601.703164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766602.535168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766602.535217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766602.727164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766603.559116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766603.751126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766604.583167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766604.583211:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583218:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766604.583223:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583234:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583248:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775766604.583249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766604.583258:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766604.583312:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076564096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766604.583338:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583341:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766604.583343:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583346:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583350:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583354:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583365:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766604.583368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076564096:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766604.583381:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076564224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766604.583390:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583412:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766604.583414:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583418:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583438:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583442:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583446:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766604.583451:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766604.583460:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076564352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766604.583469:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583472:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766604.583473:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583476:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583478:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583481:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583486:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766604.583495:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076564480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766604.583502:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766604.583506:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350930e00 x1862023076564096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766604.583506:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766604.583510:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583517:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583522:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583527:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583535:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766604.583536:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076564096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.583548:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766604.583548:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076564608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766604.583556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.583564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076564096:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766604.583565:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583572:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766604.583574:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583578:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583581:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583585:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583588:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766604.583595:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076564736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766604.583597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766604.583611:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583615:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766604.583616:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583620:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3106 00000800:00000100:0.0:1775766604.583639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766604.583645:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766604.583651:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766604.583655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350930000 x1862023076564224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766604.583658:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766604.583671:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076564864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766604.583677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076564224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766604.583681:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583689:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766604.583694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.583698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766604.583699:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766604.583703:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766604.583708:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583712:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766604.583713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564352:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766604.583719:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583724:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766604.583737:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076564992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766604.583743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766604.583752:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766604.583756:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775766604.583757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350930a80 x1862023076564352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766604.583758:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766604.583766:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3106 00000100:00080000:1.0:1775766604.583772:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766604.583777:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076564352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766604.583778:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766604.583784:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766604.583793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766604.583799:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076565120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766604.583800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.583810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766604.583820:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766604.583845:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.583857:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350932680 x1862023076564480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.583867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076564480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.583883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.583885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.583888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766604.583892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766604.583908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.583915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350931880 x1862023076564608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.583922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076564608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.583928:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.583931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.583932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076564608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766604.583948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766604.583959:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.583965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350930700 x1862023076564736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.583992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076564736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.583998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.584000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.584007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766604.584011:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766604.584025:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.584031:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350931c00 x1862023076564864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.584054:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076564864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.584084:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.584086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.584088:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766604.584093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766604.584107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.584114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350932a00 x1862023076564992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.584122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076564992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.584128:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.584130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.584132:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076564992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766604.584135:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076565120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766604.584145:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766604.584157:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766604/real 1775766604] req@ffffa04350932300 x1862023076565120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766604.584165:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076565120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766604.584170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766604.584172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766604.584174:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076565120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766604.775179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766605.607168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766605.799156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766606.631179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766606.631249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766606.823127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766607.655121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766607.847098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766608.679114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766608.679199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766608.871108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766609.703124:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703133:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775766609.703138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766609.703139:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703151:0:45069:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703163:0:45069:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703188:0:45069:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766609.703225:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076565248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766609.703257:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703260:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766609.703262:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703266:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703270:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703273:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703283:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766609.703299:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076565376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766609.703309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076565248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766609.703313:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703323:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766609.703325:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703330:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703334:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703338:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703343:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766609.703371:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076565504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766609.703381:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766609.703388:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766609.703389:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766609.703395:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703399:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703405:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703410:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703416:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766609.703426:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076565632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766609.703435:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766609.703436:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7c700 x1862023076565248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766609.703463:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766609.703466:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703475:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703479:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703482:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703486:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766609.703490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076565248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766609.703498:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076565760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766609.703531:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766609.703533:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703536:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766609.703538:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766609.703551:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766609.703553:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703561:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766609.703566:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076565248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766609.703589:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703596:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766609.703625:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076565888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766609.703634:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766609.703637:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703642:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766609.703643:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703647:0:45069:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703651:0:45069:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703655:0:45069:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703660:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766609.703666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766609.703672:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076566016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766609.703680:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766609.703682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7fb80 x1862023076565376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766609.703686:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766609.703690:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703698:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703704:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766609.703709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076565376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766609.703709:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703715:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766609.703726:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766609.703732:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076566144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766609.703733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.703736:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766609.703752:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565504:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766609.703753:0:45069:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766609.703760:0:45069:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766609.703762:0:45069:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766609.703767:0:45069:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3111 00000100:00080000:2.0:1775766609.703772:0:45069:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766609.703777:0:45069:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766609.703782:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766609.703785:0:45069:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766609.703795:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7d880 x1862023076565504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766609.703801:0:45069:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076566272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766609.703828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076565504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.703838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.703844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.703847:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766609.703856:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766609.703885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.703896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7d500 x1862023076565632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.703908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076565632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.703915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.703918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.703921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766609.703928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766609.703948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.703960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7ea00 x1862023076565760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.703976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076565760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.703989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.703995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.704000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076565760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766609.704022:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076565888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766609.704077:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.704090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7df80 x1862023076565888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.704113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076565888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.704120:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.704123:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.704125:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076565888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766609.704130:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766609.704146:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.704154:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7ce00 x1862023076566016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.704163:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076566016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.704168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.704171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.704173:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766609.704177:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766609.704187:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.704194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7ed80 x1862023076566144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.704202:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076566144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.704217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.704219:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.704222:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766609.704226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766609.704240:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766609.704246:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766609/real 1775766609] req@ffffa0434aa7c700 x1862023076566272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766609.704256:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076566272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766609.704266:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766609.704270:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766609.704273:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076566272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766609.895115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766610.727175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766610.727245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766610.919217:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766611.751146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766611.943193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766612.775201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766612.775204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766612.967154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766613.799150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766613.991216:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766614.823124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766614.823193:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766614.823202:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766614.823207:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775766614.823216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766614.823218:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823230:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823237:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766614.823287:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076566400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766614.823345:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766614.823350:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766614.823351:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766614.823355:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823359:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823363:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823375:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766614.823406:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076566528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766614.823407:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076566400:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766614.823424:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766614.823428:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766614.823431:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766614.823436:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823440:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823444:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823451:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766614.823482:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076566656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766614.823493:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766614.823494:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766614.823500:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766614.823502:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766614.823507:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823535:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823540:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823546:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766614.823553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349640e00 x1862023076566400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766614.823560:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076566784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766614.823574:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766614.823577:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766614.823579:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766614.823581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076566400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766614.823584:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823587:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823594:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823600:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766614.823607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766614.823611:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076566912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766614.823614:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.823623:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076566400:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766614.823626:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766614.823630:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766614.823631:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766614.823635:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823637:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823641:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823644:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766614.823651:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076567040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766614.823653:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766614.823681:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766614.823683:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766614.823694:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766614.823698:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766614.823698:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349641c00 x1862023076566528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766614.823707:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823715:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766614.823718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076566528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766614.823722:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823731:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766614.823733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.823740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.823745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766614.823747:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076567168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766614.823758:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766614.823760:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766614.823761:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766614.823765:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766614.823773:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:0.0:1775766614.823777:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766614.823781:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823787:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766614.823795:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766614.823804:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076567296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766614.823812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349640700 x1862023076566656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766614.823823:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766614.823828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076566656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766614.823830:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766614.823835:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766614.823842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766614.823846:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3116 00000100:00080000:1.0:1775766614.823849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.823851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766614.823859:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766614.823866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566784:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766614.823869:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766614.823879:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775766614.823891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766614.823898:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076567424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766614.823902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349642680 x1862023076566784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.823927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076566784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.823934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.823937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.823940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766614.823945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766614.823957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766614.823963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349640a80 x1862023076566912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.823970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076566912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.823976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.823978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.823979:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076566912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766614.823995:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766614.824005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766614.824012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349640e00 x1862023076567040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.824047:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076567040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.824054:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.824056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.824058:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766614.824063:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766614.824078:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766614.824085:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349643100 x1862023076567168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.824094:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076567168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.824099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.824102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.824104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766614.824107:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766614.824116:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766614.824121:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349641180 x1862023076567296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.824128:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076567296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.824142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.824154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.824156:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766614.824160:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766614.824175:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766614.824181:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766614/real 1775766614] req@ffffa04349641500 x1862023076567424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766614.824188:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076567424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766614.824194:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766614.824196:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766614.824198:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076567424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766615.015166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766615.847170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766616.039128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766616.871100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766616.871291:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766617.063148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766617.895108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766618.087133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766618.919097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766618.919197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766619.111118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766619.943119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766619.943123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766619.943137:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766619.943150:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:0.0:1775766619.943163:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943174:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766619.943222:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076567552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766619.943249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766619.943255:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766619.943260:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:0.0:1775766619.943264:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943269:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766619.943295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076567552:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766619.943313:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076567680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766619.943324:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766619.943329:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766619.943333:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:0.0:1775766619.943336:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943340:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943345:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766619.943354:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076567808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766619.943362:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766619.943367:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766619.943371:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3121 00000800:00000100:1.0:1775766619.943371:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766619.943375:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943378:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943383:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766619.943392:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076567936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766619.943400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766619.943406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766619.943415:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349642300 x1862023076567552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766619.943416:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:0.0:1775766619.943421:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943425:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943429:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766619.943435:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076567552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766619.943440:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076568064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766619.943451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766619.943452:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766619.943454:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766619.943456:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766619.943457:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766619.943460:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00100000:1.0:1775766619.943461:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076567552:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766619.943479:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943483:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943488:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766619.943498:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076568192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766619.943503:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766619.943509:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766619.943513:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766619.943518:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:0.0:1775766619.943523:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766619.943526:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766619.943527:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766619.943533:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766619.943539:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349643800 x1862023076567680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766619.943542:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076568320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766619.943552:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076567680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766619.943556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766619.943560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766619.943561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766619.943562:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766619.943564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766619.943566:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00100000:1.0:1775766619.943568:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766619.943570:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766619.943574:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943574:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567808:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766619.943579:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766619.943590:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076568448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766619.943592:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766619.943600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766619.943602:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349642a00 x1862023076567808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766619.943603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766619.943605:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766619.943614:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076567808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766619.943620:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3121 00000100:00080000:1.0:1775766619.943623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766619.943624:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766619.943626:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766619.943629:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567808:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766619.943634:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766619.943642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766619.943643:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076568576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766619.943666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349640380 x1862023076567936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076567936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.943702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.943705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076567936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766619.943712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766619.943731:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349641880 x1862023076568064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076568064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.943757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.943759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766619.943780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766619.943800:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349643b80 x1862023076568192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076568192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.943844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.943847:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766619.943851:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766619.943866:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349642300 x1862023076568320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076568320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.943889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.943891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766619.943894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766619.943903:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349641500 x1862023076568448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076568448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.943942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.943946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766619.943952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766619.943974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766619.943984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766619/real 1775766619] req@ffffa04349641180 x1862023076568576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766619.943992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076568576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766619.943998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766619.944000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766619.944002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076568576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766620.135155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766620.967142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766620.967160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766621.159142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766621.991092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766622.183131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766623.015138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766623.015180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766623.207173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766624.039178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766624.231165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766625.063174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766625.063249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766625.063382:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076568704:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775766625.063516:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.063580:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825ce00 x1862023076568704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.063640:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076568704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.063801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.063816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.063823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076568704:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766625.063858:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766625.063898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.063915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825d880 x1862023076568832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.063931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076568832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.063939:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.063942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.063944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766625.063949:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766625.063963:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.063969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825dc00 x1862023076568960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.063977:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076568960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.063983:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.063985:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076568960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766625.064015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766625.064032:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064063:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825d500 x1862023076569088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064075:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076569088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064081:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064084:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766625.064091:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766625.064105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825f100 x1862023076569216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064120:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076569216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064127:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064129:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766625.064135:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766625.064163:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064172:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825e300 x1862023076569344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076569344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064199:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064202:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064204:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766625.064209:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766625.064226:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064234:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825ed80 x1862023076569472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064244:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076569472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064251:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064258:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064266:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766625.064272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766625.064289:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064297:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825c000 x1862023076569600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064307:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076569600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064325:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064329:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064332:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766625.064366:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766625.064384:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766625.064393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766625/real 1775766625] req@ffffa0435825e680 x1862023076569728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766625.064403:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076569728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766625.064429:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766625.064433:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766625.064435:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076569728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766625.255172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766626.087121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766626.279119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766627.111125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766627.111210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766627.303128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766628.135133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766628.135148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766628.327136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766629.159121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766629.159180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766629.351121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766630.183123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766630.183153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766630.183159:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183167:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766630.183172:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183182:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183192:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183198:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766630.183242:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076569856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766630.183270:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183274:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766630.183276:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183281:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183285:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183291:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183302:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766630.183311:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076569856:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766630.183318:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076569984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766630.183378:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766630.183416:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766630.183422:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766630.183425:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183431:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183443:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183447:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183454:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766630.183462:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349643100 x1862023076569856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766630.183469:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076570112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766630.183490:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183493:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766630.183495:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766630.183499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076569856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766630.183500:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183503:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183506:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183509:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766630.183520:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076570240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766630.183525:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766630.183529:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183533:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766630.183535:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766630.183535:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766630.183542:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00100000:1.0:1775766630.183545:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076569856:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766630.183553:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183558:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183561:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766630.183569:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076570368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766630.183573:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766630.183587:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183590:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766630.183592:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183596:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183599:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183601:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183604:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766630.183616:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076570496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766630.183616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766630.183626:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183629:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766630.183630:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183633:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183635:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766630.183639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349640e00 x1862023076569984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766630.183640:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183645:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766630.183655:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076570624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766630.183663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076569984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766630.183684:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183688:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766630.183689:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766630.183690:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766630.183694:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183699:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766630.183701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766630.183705:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766630.183710:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076569984:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766630.183711:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766630.183724:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076570752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766630.183725:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570112:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766630.183739:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766630.183742:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766630.183743:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766630.183746:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3131 00000100:00080000:0.0:1775766630.183753:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766630.183757:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766630.183759:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766630.183762:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766630.183771:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076570880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766630.183778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349640a80 x1862023076570112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.183793:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076570112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.183803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.183807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.183811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766630.183823:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766630.183843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.183853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349642680 x1862023076570240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.183865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076570240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.183874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.183878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.183881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766630.183899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766630.183916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.183926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349640700 x1862023076570368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.183938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076570368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.183947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.183951:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.183954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076570368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766630.183959:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766630.183975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.183985:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349641c00 x1862023076570496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.184003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076570496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.184012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.184016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.184019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766630.184025:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766630.184066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.184077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349643100 x1862023076570624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.184090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076570624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.184098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.184102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.184105:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766630.184111:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766630.184129:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.184138:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349641180 x1862023076570752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.184150:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076570752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.184171:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.184175:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.184178:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766630.184185:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766630.184203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766630.184213:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766630/real 1775766630] req@ffffa04349641500 x1862023076570880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766630.184225:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076570880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766630.184233:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766630.184237:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766630.184240:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076570880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766630.375143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766631.207201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766631.207250:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766631.399150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766631.399197:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766632.231197:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766632.231323:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766632.423122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766632.423181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766633.255120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766633.255245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766633.447191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766633.447191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766634.279192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766634.279208:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766634.471108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766634.471134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766635.303123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766635.303128:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303137:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766635.303141:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303152:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3136 00010000:00010000:1.0:1775766635.303156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766635.303163:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303171:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766635.303218:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076571008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303256:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303259:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766635.303260:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303263:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303266:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303270:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303280:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766635.303290:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076571008:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766635.303291:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076571136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303301:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303317:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766635.303318:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303322:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303324:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303327:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303351:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766635.303361:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076571264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303368:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303372:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766635.303373:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303376:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303379:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303383:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303386:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766635.303395:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076571392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303400:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303404:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766635.303405:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303409:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303412:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303415:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303418:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766635.303427:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076571136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766635.303428:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076571520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303440:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303442:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766635.303448:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303452:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303456:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766635.303461:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076571264:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766635.303469:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303477:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766635.303490:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076571648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303498:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766635.303501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076571392:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766635.303503:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766635.303507:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303512:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303516:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303519:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303523:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766635.303531:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076571520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766635.303536:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076571776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303545:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303547:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766635.303549:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766635.303552:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00100000:1.0:1775766635.303555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076571648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766635.303557:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303562:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303566:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766635.303578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076571776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766635.303585:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076571904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766635.303594:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766635.303597:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766635.303599:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766635.303602:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076571904:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766635.303606:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3136 00000100:00080000:0.0:1775766635.303613:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766635.303616:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766635.303620:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766635.303628:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076572032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766635.303642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076572032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766635.495144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766635.495147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766636.327158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766636.327253:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766636.519077:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766636.519102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766637.351113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766637.351231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766637.543123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766637.543258:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766638.375389:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766638.375410:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775766638.375451:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775766638.375510:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775766638.375557:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375642:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375670:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375693:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375717:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375738:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375778:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375821:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766638.375861:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775766638.567121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766638.567203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766639.399124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766639.399227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766639.591115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766639.591142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766640.423110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766640.423131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766640.423143:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423153:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423162:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423167:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766640.423194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766640.423198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076572160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766640.423227:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423231:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423235:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423274:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766640.423304:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076572288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766640.423322:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423329:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766640.423350:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076572416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766640.423370:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423375:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423382:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423386:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766640.423411:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076572544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423436:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766640.423439:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423446:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423454:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423462:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766640.423488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076572672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766640.423507:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423515:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423521:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423526:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423532:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766640.423554:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076572800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766640.423582:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423591:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423600:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423606:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423624:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766640.423640:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076572928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423655:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766640.423657:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423665:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423670:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423675:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423687:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766640.423709:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076573056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766640.423723:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766640.423732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766640.423737:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766640.423745:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3142 00000100:00080000:1.0:1775766640.423752:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766640.423760:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766640.423769:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766640.423790:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076573184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766640.615120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766640.615120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766641.447147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766641.447328:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766641.639089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766641.639110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766642.471119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766642.471164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766642.663127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766642.663143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766643.495162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766643.495203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766643.687102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766643.687130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766644.519116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766644.519260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766644.711146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766644.711146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766645.543157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766645.543339:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076573312:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766645.543411:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543455:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825ea00 x1862023076573312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.543473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076573312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.543482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.543486:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.543489:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076573312:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766645.543508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766645.543525:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543534:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825e680 x1862023076573440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.543544:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076573440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.543560:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.543567:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.543572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766645.543580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766645.543604:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825c000 x1862023076573568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.543653:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076573568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.543659:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.543662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.543664:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766645.543667:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766645.543682:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825ed80 x1862023076573696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.543698:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076573696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.543704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.543706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.543708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766645.543723:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766645.543738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825e300 x1862023076573824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.543753:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076573824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.543773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.543779:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.543783:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076573824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766645.543872:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076573952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766645.543916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.543965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825f100 x1862023076573952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.544008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076573952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.544038:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.544091:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.544105:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076573952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766645.544165:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766645.544222:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.544252:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825d500 x1862023076574080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.544265:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076574080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.544272:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.544275:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00010000:00010000:2.0:1775766645.544278:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766645.544289:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766645.544298:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766645.544321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.544333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825dc00 x1862023076574208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.544346:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076574208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.544355:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.544358:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.544362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766645.544376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766645.544392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766645.544399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766645/real 1775766645] req@ffffa0435825d880 x1862023076574336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766645.544407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076574336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766645.544413:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766645.544415:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766645.544417:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076574336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766645.735168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766645.735175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766646.567139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766646.567216:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766646.759111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766646.759111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766647.591158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766647.591221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766647.783159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766647.783161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766648.615238:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766648.615334:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766648.807132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766648.807159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766649.639200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766649.639261:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766649.831112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766649.831112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766650.663114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766650.663260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076574464:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766650.663322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663370:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825ce00 x1862023076574464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663389:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076574464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663412:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663422:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076574464:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766650.663453:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766650.663484:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663502:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825d880 x1862023076574592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076574592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663551:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766650.663563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766650.663605:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663621:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825dc00 x1862023076574720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076574720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663650:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663654:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663659:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766650.663666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574848:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775766650.663672:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1775766650.663706:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825d500 x1862023076574848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663727:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076574848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766650.663743:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766650.663756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825f100 x1862023076574976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076574976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663811:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076574976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766650.663826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766650.663850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825e300 x1862023076575104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663893:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076575104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.663906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.663908:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766650.663927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766650.663950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.663962:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825ed80 x1862023076575232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.663976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076575232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.663998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.664001:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.664003:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766650.664008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766650.664024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.664033:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825c000 x1862023076575360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.664044:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076575360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.664053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.664056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.664059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766650.664065:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766650.664117:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766650.664126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766650/real 1775766650] req@ffffa0435825e680 x1862023076575488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766650.664137:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076575488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766650.664162:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766650.664166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766650.664177:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076575488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766650.855204:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766650.855208:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775766651.687229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766651.687247:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766651.879095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766651.879096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766652.711154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766652.711195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766652.903155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766652.903156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766653.735139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766653.735200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766653.927100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766653.927121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766654.759158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766654.759190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766654.951153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766654.951155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766655.783150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766655.783299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076575616:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766655.783355:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825ea00 x1862023076575616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076575616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783426:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783432:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783439:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076575616:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766655.783463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076575744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766655.783497:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825e680 x1862023076575744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076575744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783566:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783573:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783578:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076575744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766655.783588:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076575872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766655.783612:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825c000 x1862023076575872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076575872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783676:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076575872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766655.783693:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766655.783709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825ed80 x1862023076576000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076576000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783740:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783746:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766655.783764:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766655.783788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825e300 x1862023076576128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783851:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076576128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766655.783885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766655.783918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.783935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825f100 x1862023076576256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.783950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076576256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.783957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.783960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.783963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766655.783978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766655.783999:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.784015:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825d500 x1862023076576384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1775766655.784023:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766655.784045:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076576384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.784052:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.784055:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.784059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766655.784065:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766655.784081:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.784102:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825dc00 x1862023076576512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.784113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076576512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.784119:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.784121:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.784123:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766655.784127:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766655.784140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766655.784146:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766655/real 1775766655] req@ffffa0435825d880 x1862023076576640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766655.784153:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076576640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766655.784163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766655.784165:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766655.784167:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076576640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766655.975109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766655.975111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766656.807169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766656.807239:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766656.999144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766656.999146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766657.831110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766657.831183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766658.023087:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766658.023124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766658.855110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766658.855176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766659.047125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766659.047127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766659.879111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766659.879285:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766660.071118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766660.071121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766660.903184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766660.903192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903199:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766660.903204:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903214:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903227:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903253:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766660.903291:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076576768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766660.903321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903329:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766660.903352:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076576768:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766660.903355:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076576896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766660.903385:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903405:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903409:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903413:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766660.903428:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076577024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766660.903439:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903442:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903444:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903447:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903450:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775766660.903454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766660.903462:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076577152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766660.903475:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903477:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903488:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903491:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903494:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766660.903502:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350930e00 x1862023076576768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766660.903512:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076577280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766660.903531:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903537:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00100000:0.0:1775766660.903538:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076576768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766660.903540:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903556:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766660.903561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766660.903566:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076577408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766660.903568:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.903576:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076576768:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766660.903584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766660.903589:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903592:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00080000:1.0:1775766660.903595:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903598:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766660.903607:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766660.903610:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076577536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766660.903620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766660.903623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766660.903624:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766660.903635:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3162 00000800:00000100:0.0:1775766660.903639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766660.903642:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903647:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903655:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766660.903665:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350930000 x1862023076576896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766660.903673:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076577664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766660.903683:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076576896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766660.903697:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766660.903699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.903706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766660.903706:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766660.903711:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766660.903715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076576896:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766660.903722:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3162 00000100:00100000:0.0:1775766660.903725:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766660.903728:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766660.903734:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766660.903739:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766660.903750:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076577792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766660.903757:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.903780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350930a80 x1862023076577024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.903797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076577024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.903803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.903806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.903809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766660.903814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766660.903831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.903838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350932680 x1862023076577152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.903847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076577152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.903858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.903860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.903863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766660.903867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766660.903881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.903888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350931880 x1862023076577280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.903896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076577280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.903902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.903904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.903906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076577280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766660.903910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766660.903921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.903927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350930700 x1862023076577408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.903935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076577408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.903940:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.903942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.903950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766660.903972:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766660.903996:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.904008:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350931c00 x1862023076577536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.904022:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076577536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.904030:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.904032:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.904070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766660.904076:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766660.904094:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.904102:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350932a00 x1862023076577664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.904111:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076577664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.904117:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.904119:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.904121:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766660.904125:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766660.904136:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766660.904150:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766660/real 1775766660] req@ffffa04350932300 x1862023076577792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766660.904158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076577792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766660.904164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766660.904166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766660.904168:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076577792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766661.095103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766661.095167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766661.927130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766661.927185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766662.119111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766662.119112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766662.951177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766663.143132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766663.143132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766663.975154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766663.975239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766664.167105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766664.167115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766664.999215:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766665.191117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766665.191117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766666.023160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766666.023183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766666.023209:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766666.023236:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023248:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023261:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023285:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766666.023405:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076577920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766666.023437:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766666.023442:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023446:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023450:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023453:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023463:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766666.023473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076578048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766666.023481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766666.023485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023488:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023494:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023498:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766666.023523:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076578176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766666.023533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076577920:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766666.023554:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023558:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766666.023559:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023588:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023592:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023597:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766666.023614:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076578304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766666.023624:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766666.023626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023630:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766666.023632:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023636:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023641:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023647:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023652:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766666.023661:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076578432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766666.023670:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350932300 x1862023076577920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766666.023674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766666.023698:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023710:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023714:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766666.023720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076577920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766666.023724:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023738:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766666.023749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.023758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766666.023758:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076578560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766666.023781:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766666.023783:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076577920:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766666.023790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766666.023797:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766666.023804:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00100000:0.0:1775766666.023807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766666.023812:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.023818:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.023824:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766666.023837:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076578688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766666.023842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766666.023861:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766666.023867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350932a00 x1862023076578048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766666.023873:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766666.023878:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766666.023888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076578048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766666.023903:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.023916:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766666.023918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766666.023926:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766666.023932:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775766666.023936:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766666.023943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766666.023956:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076578816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766666.023964:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766666.023967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766666.023969:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766666.023974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766666.023998:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3167 00000100:00080000:1.0:1775766666.024011:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766666.024016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766666.024023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775766666.024080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766666.024088:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076578944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766666.024096:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350931c00 x1862023076578176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024119:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076578176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024126:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766666.024139:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766666.024153:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024169:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350930700 x1862023076578304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024177:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076578304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766666.024191:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766666.024203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024208:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350931880 x1862023076578432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076578432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024221:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024239:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024241:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076578432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766666.024245:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766666.024256:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024272:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350932680 x1862023076578560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024279:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076578560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024304:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024309:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766666.024313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766666.024326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350930a80 x1862023076578688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024340:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076578688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024345:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024347:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024349:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766666.024353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766666.024363:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024368:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350930000 x1862023076578816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076578816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024408:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024413:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766666.024490:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766666.024507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766666.024515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766666/real 1775766666] req@ffffa04350930e00 x1862023076578944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766666.024523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076578944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766666.024539:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766666.024541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766666.024543:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076578944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766666.215094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766666.215096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766667.047125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766667.239124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766667.239124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766668.071128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766668.071192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766668.263111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766668.263132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766669.095125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766669.287122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766669.287137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766670.119143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766670.119171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766670.311104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766670.311113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766671.143155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766671.143350:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076579072:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775766671.143424:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.143486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825ce00 x1862023076579072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.143513:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076579072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.143535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.143555:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.143561:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076579072:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766671.143600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766671.143634:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.143651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825d880 x1862023076579200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.143672:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076579200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.143699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.143706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.143715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766671.143729:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766671.143761:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.143782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825dc00 x1862023076579328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.143804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076579328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.143817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.143821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.143826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766671.143832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766671.143877:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.143887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825d500 x1862023076579456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.143898:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076579456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.143906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.143909:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.143912:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766671.143920:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766671.143967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.143980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825f100 x1862023076579584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.144007:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076579584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.144016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.144020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.144025:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076579584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766671.144099:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766671.144126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.144140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825e300 x1862023076579712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.144188:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076579712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.144196:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.144198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.144201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766671.144206:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766671.144225:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.144233:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825ed80 x1862023076579840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.144243:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076579840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.144250:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.144252:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.144255:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766671.144259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766671.144271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.144277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825c000 x1862023076579968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.144286:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076579968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.144319:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.144321:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.144324:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076579968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766671.144328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076580096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766671.144343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766671.144351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766671/real 1775766671] req@ffffa0435825e680 x1862023076580096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766671.144359:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076580096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766671.144365:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766671.144367:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766671.144369:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076580096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766671.335150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766671.335159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766672.167122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766672.167222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766672.359183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766672.359185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766673.191182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766673.383126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766673.383164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766674.215169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766674.407159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766674.407174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766675.239165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766675.431122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766675.431170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766676.263172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766676.263227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766676.263243:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263253:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3177 00010000:00010000:1.0:1775766676.263277:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766676.263284:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263292:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766676.263338:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076580224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766676.263371:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263375:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263379:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263384:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263394:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766676.263407:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076580352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766676.263425:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263429:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263439:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263443:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766676.263458:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076580480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766676.263485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263490:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263520:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263525:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263531:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766676.263546:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076580608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263561:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766676.263563:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263567:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263575:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263583:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263593:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766676.263612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076580736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263624:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263632:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766676.263638:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263651:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263668:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263677:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766676.263718:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076580864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766676.263770:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263778:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263791:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263801:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263810:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766676.263860:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076580992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.263898:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.263906:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766676.263910:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.263932:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.263939:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.263944:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.263953:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766676.263978:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076581120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766676.264016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766676.264023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766676.264070:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766676.264077:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3177 00000100:00080000:0.0:1775766676.264083:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766676.264088:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766676.264095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766676.264131:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076581248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766676.455137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766676.455137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766677.287249:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766677.479119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766677.479119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766678.311203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766678.311305:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766678.503114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766678.503134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766679.335208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766679.335264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766679.527130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766679.527196:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766680.359208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766680.359352:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766680.551161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766680.551161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766681.383182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766681.383185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766681.383210:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383227:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383244:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383257:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766681.383319:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076581376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766681.383376:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383383:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383391:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383423:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766681.383473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076581504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383504:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766681.383509:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383517:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383525:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383532:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383541:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766681.383559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076581632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766681.383581:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383586:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383591:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383596:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383603:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766681.383618:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076581760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383642:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766681.383647:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383659:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383666:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383670:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383676:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766681.383686:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076581888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766681.383701:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383704:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383707:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383710:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383713:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766681.383723:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076582016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383735:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766681.383737:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383740:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383743:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383745:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383758:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766681.383768:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076582144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383778:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383782:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766681.383785:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383791:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383799:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383806:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383814:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766681.383870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076582272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766681.383883:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766681.383887:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766681.383890:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766681.383895:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3183 00000100:00080000:0.0:1775766681.383900:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766681.383905:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766681.383922:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766681.383942:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076582400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766681.575146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766681.575169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766682.407155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766682.407187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766682.599116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766682.599116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766683.431129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766683.623129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766683.623163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766684.455105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766684.455254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766684.647096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766684.647183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766685.479103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766685.671090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766685.671123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766686.503126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766686.503126:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766686.503139:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503147:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3188 00010000:00010000:0.0:1775766686.503160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766686.503167:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503174:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766686.503203:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076582528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503224:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503226:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766686.503228:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503231:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503233:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503236:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503243:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766686.503252:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076582656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503262:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503265:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766686.503267:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503270:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503272:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503274:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503278:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766686.503285:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076582784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503292:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503299:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766686.503300:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503303:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503307:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503309:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503313:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766686.503324:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076582912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503332:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766686.503336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503339:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503341:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503344:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503347:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766686.503356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076583040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503375:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766686.503390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503396:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503403:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503411:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503424:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766686.503444:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076583168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503461:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503469:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766686.503473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503477:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503483:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766686.503511:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076583296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503518:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503523:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766686.503526:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503529:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503531:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503533:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503538:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766686.503550:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076583424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766686.503564:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766686.503570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766686.503580:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766686.503592:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3188 00000100:00080000:2.0:1775766686.503601:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766686.503608:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766686.503613:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766686.503628:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076583552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766686.695126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766686.695132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766687.527103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766687.719081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766687.719084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766688.551143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766688.551168:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766688.743108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766688.743121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766689.575169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766689.767126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766689.767167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766690.599166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766690.599211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766690.791096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766690.791107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766691.623131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766691.623146:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623154:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766691.623167:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623178:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623190:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623198:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766691.623242:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076583680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623267:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623271:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766691.623292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623296:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623302:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623310:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766691.623334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076583808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623343:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623346:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766691.623347:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623351:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623353:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623355:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766691.623367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076583936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623379:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766691.623383:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623386:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623390:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623393:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623413:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766691.623426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076584064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623434:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623436:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766691.623438:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623446:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623454:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623460:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766691.623470:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076584192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623492:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623496:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766691.623498:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623506:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623512:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623518:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623521:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766691.623530:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076584320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623536:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623541:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766691.623544:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623554:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623558:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623562:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623571:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766691.623585:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076584448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623604:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623611:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766691.623614:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623621:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623625:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623632:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623638:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766691.623648:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076584576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766691.623659:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766691.623664:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766691.623665:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766691.623670:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3193 00000100:00080000:1.0:1775766691.623673:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766691.623676:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766691.623680:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766691.623694:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076584704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766691.815174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766691.815184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766692.647144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766692.647210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766692.839119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766692.839126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766693.671096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766693.863133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766693.863133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766694.695114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766694.695165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766694.887094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766694.887094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766695.719126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766695.911114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766695.911115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766696.743097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766696.743108:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743114:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766696.743119:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743128:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743139:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743147:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775766696.743153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766696.743187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076584832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743209:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743212:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766696.743213:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743217:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743220:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743224:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743232:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766696.743245:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076584960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743257:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766696.743261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743263:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743266:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743268:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766696.743300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076585088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766696.743319:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743325:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743331:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743336:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743342:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766696.743352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076585216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743359:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766696.743363:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743366:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743370:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743374:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743378:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766696.743389:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076585344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743399:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743402:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766696.743410:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743418:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743424:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743430:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743434:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766696.743446:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076585472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743456:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766696.743463:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743467:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743469:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743471:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743473:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766696.743479:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076585600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743488:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766696.743496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743499:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743501:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743503:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743507:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766696.743522:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076585728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766696.743530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766696.743533:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766696.743534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766696.743537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3198 00000100:00080000:1.0:1775766696.743540:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766696.743545:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766696.743550:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766696.743560:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076585856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766696.935077:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766696.935106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766697.767126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766697.959187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766697.959203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766698.791163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766698.791232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766698.983114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766698.983147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766699.815114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766700.007187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766700.007226:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766700.839163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766700.839260:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766701.031131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766701.031175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766701.863129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766701.863130:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863140:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766701.863146:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863156:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863168:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863177:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766701.863219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076585984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863249:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863254:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766701.863256:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863275:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863282:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863291:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863304:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766701.863323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076586112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863351:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863356:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766701.863359:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863364:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863371:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863378:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766701.863410:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076586240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863439:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863446:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766701.863450:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863459:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863467:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863486:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766701.863520:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076586368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863583:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766701.863586:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863593:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863638:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863646:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863672:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766701.863691:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076586496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863705:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863710:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766701.863713:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863719:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863728:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863735:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863743:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766701.863758:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076586624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863777:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863782:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766701.863784:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863790:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863817:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863825:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863833:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766701.863860:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076586752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863880:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863885:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766701.863887:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863893:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863898:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863906:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863916:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766701.863931:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076586880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766701.863945:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766701.863949:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766701.863951:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766701.863960:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3203 00000100:00080000:1.0:1775766701.863968:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766701.863974:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766701.863981:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766701.863994:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076587008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766702.055116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766702.055121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766702.887236:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766703.079123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766703.079123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766703.911163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766704.103109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766704.103110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766704.935113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766704.935818:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350932300 x1862023076587008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.935846:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076587008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.935872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.935878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.935884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076587008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766704.935894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350932a00 x1862023076586880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.935909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076586880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.935926:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.935931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.935935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076586880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766704.935943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350931c00 x1862023076586752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.935957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076586752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.935975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.935979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.935989:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076586752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766704.935997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350930700 x1862023076586624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076586624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076586624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766704.936161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350931880 x1862023076586496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076586496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936205:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936210:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076586496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766704.936221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350932680 x1862023076586368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076586368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936250:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936254:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936258:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076586368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766704.936265:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350930a80 x1862023076586240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076586240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936402:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936406:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936410:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076586240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766704.936420:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350930000 x1862023076586112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936432:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076586112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936441:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936446:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936450:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076586112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766704.936456:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766701/real 1775766704] req@ffffa04350930e00 x1862023076585984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766704.936470:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076585984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766704.936487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766704.936491:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766704.936495:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076585984:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766705.127110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766705.127137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766705.959109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766706.151126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766706.151141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775766706.983103:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766706.983104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766706.983112:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766706.983116:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983123:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983133:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983140:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766706.983171:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076587136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983195:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983199:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766706.983200:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983204:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983208:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983211:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983218:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766706.983232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076587264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983240:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766706.983245:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983249:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983252:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983270:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766706.983282:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076587392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983299:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766706.983302:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983311:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983316:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983321:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766706.983346:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076587520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983356:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983360:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766706.983362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983366:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766706.983398:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076587648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983415:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766706.983437:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983445:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983450:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983460:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983468:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766706.983487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076587776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983512:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766706.983515:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983521:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983527:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983535:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983541:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766706.983568:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076587904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983580:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983588:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766706.983593:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983601:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983607:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983619:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766706.983646:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076588032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766706.983663:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766706.983667:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766706.983670:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766706.983676:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3208 00000100:00080000:1.0:1775766706.983682:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766706.983691:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766706.983698:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766706.983712:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076588160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766707.175100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766707.175100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766708.007266:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766708.199099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766708.199099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766709.031138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766709.223109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766709.223124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766710.055119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766710.247104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766710.247171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766711.079233:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766711.271121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766711.271123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766712.103228:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766712.103268:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103280:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766712.103285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103301:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103318:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103328:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766712.103371:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076588288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766712.103411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103415:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103418:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103421:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.103431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766712.103463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076588416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103520:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103526:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766712.103529:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103548:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103555:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103563:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.103573:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766712.103596:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076588544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103609:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103613:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766712.103615:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103619:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103625:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103629:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.103635:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766712.103652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076588672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103663:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103668:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766712.103671:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103687:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103692:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103696:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.103703:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766712.103729:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076588800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103809:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103813:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766712.103814:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103833:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103840:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.103852:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.103879:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766712.103913:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076588928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.103928:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.103934:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766712.103938:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.103978:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.103998:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.104011:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.104020:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766712.104069:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076589056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.104089:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.104092:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766712.104094:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.104098:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.104102:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.104121:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.104143:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766712.104166:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076589184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766712.104260:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766712.104277:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766712.104285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766712.104303:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3213 00000100:00080000:2.0:1775766712.104317:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766712.104324:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766712.104336:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766712.104363:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076589312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766712.295129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766712.295143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766713.127107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766713.319135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766713.319135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766714.151141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766714.343128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766714.343128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766715.175107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766715.367143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766715.367144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766716.199141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766716.391101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766716.391124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766717.223190:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766717.223192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766717.223201:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766717.223205:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223217:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223241:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223248:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766717.223278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076589440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223305:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766717.223310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223322:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223334:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766717.223347:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076589568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223356:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223359:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766717.223361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223364:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223468:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223472:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223477:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766717.223489:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076589696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223497:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223506:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766717.223508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223512:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223515:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223518:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766717.223531:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076589824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766717.223541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223543:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223545:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223547:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223549:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766717.223556:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076589952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223562:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223564:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766717.223566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223569:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223571:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223573:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223581:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766717.223589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076590080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223597:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223599:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766717.223601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223606:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223608:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223615:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223624:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766717.223633:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076590208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223640:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223643:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766717.223644:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223646:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223648:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223650:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223654:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766717.223660:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076590336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766717.223668:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766717.223670:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766717.223676:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766717.223683:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3218 00000100:00080000:0.0:1775766717.223690:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766717.223698:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766717.223717:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766717.223735:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076590464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766717.415149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766717.415149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766718.247172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766718.439151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766718.439191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766719.271191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766719.271247:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766719.463118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766719.463118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766720.295132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766720.487097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766720.487102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766721.319137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766721.319173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766721.511096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766721.511109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766722.343143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766722.343191:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343200:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766722.343206:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343218:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343229:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343236:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766722.343280:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076590592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766722.343319:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343324:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343328:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343333:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343344:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766722.343359:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076590720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343386:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343390:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766722.343412:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343438:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343445:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343451:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343458:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766722.343476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076590848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343498:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766722.343505:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343511:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343516:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343521:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343528:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766722.343545:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076590976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343587:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766722.343592:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343600:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343610:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343650:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343666:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766722.343700:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076591104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343741:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343747:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766722.343750:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343758:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343772:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343783:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343792:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766722.343826:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076591232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343876:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343887:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766722.343892:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343905:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343916:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.343922:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.343932:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766722.343950:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076591360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.343966:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.343971:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766722.343973:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.343979:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.343996:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.344002:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.344014:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766722.344066:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076591488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766722.344078:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766722.344082:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766722.344085:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766722.344089:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3224 00000100:00080000:2.0:1775766722.344094:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766722.344098:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766722.344103:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766722.344142:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076591616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766722.535109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766722.535109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766723.367128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766723.367186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766723.559131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766723.559131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766724.391120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766724.583107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766724.583110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766725.415180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766725.415298:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766725.607140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766725.607227:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766726.439179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766726.631124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766726.631128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766727.463156:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766727.463161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766727.463170:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766727.463177:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463190:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463221:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463227:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766727.463268:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076591744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463308:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766727.463313:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463317:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463320:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463325:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463335:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766727.463381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076591872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766727.463411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463416:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463420:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463423:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463428:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766727.463437:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076592000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463451:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463459:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766727.463461:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463464:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463468:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463475:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463482:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766727.463495:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076592128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766727.463506:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463510:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463517:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463522:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766727.463533:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076592256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463571:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463575:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766727.463577:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463583:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463596:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463609:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463620:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766727.463662:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076592384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463672:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463677:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766727.463679:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463689:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463700:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463706:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463710:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766727.463728:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076592512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463750:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463755:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766727.463760:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463771:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463779:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463786:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463790:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766727.463798:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076592640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766727.463860:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766727.463868:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766727.463881:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766727.463899:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3229 00000100:00080000:2.0:1775766727.463915:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766727.463923:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766727.463929:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766727.463944:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076592768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766727.655108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766727.655110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766728.487286:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766728.679105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766729.511190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766729.703099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766730.535161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766730.727122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766731.559162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766731.751127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766732.583142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766732.583155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766732.583171:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583183:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3234 00010000:00010000:1.0:1775766732.583191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766732.583198:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583207:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766732.583256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076592896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766732.583304:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583308:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583335:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766732.583360:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076593024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766732.583374:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583377:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583381:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583383:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583387:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766732.583396:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076593152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766732.583407:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583410:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583415:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583418:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583422:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766732.583429:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076593280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583448:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766732.583450:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583455:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583460:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583463:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583466:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766732.583475:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076593408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583482:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766732.583487:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583494:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583500:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583507:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583510:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766732.583520:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076593536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766732.583548:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583554:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583558:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583563:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583578:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766732.583589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076593664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583595:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766732.583600:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583606:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583612:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583615:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583618:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766732.583628:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076593792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766732.583641:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766732.583645:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766732.583647:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766732.583651:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3234 00000100:00080000:2.0:1775766732.583656:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766732.583662:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766732.583669:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766732.583681:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076593920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766732.583774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7f480 x1862023076593280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.583790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076593280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.583796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.583798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.583800:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076593280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766732.583803:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076593408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766732.583816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766732.583822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7dc00 x1862023076593408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.583829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076593408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.583835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.583837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.583839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076593408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766732.583842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766732.583852:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766732.583858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7c000 x1862023076593536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.583865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076593536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.583881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.583883:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.583886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766732.583889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766732.583903:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766732.583910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7f100 x1862023076593664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.583917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076593664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.583923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.583924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.583926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766732.583940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766732.583951:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766732.583957:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7f800 x1862023076593792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.583965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076593792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.583970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.583972:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.583979:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766732.583983:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766732.583996:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766732.584003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766732/real 1775766732] req@ffffa0434aa7e300 x1862023076593920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766732.584010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076593920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766732.584016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766732.584017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766732.584019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076593920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766732.775158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766732.775159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766733.607220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766733.607292:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766733.799114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766733.799114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766734.631161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766734.631259:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766734.823109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766734.823132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766735.655096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766735.655125:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766735.847115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766735.847177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766736.679179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766736.679305:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766736.871172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766736.871175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766737.703131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766737.703142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766737.703144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766737.703153:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703167:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703180:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775766737.703183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766737.703193:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766737.703268:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076594048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766737.703297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766737.703302:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703306:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703315:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703324:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766737.703336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076594176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766737.703348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766737.703354:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076594048:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766737.703355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766737.703360:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703386:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703393:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703399:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703407:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766737.703426:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076594304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766737.703433:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766737.703446:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766737.703452:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703457:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703463:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703466:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703471:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766737.703483:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076594432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766737.703491:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766737.703494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7c700 x1862023076594048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766737.703495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766737.703498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703507:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703513:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703520:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703524:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766737.703534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076594048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766737.703536:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076594560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766737.703569:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766737.703577:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766737.703578:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766737.703582:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:3.0:1775766737.703588:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766737.703589:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766737.703596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076594048:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766737.703597:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703607:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766737.703619:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076594688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766737.703627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766737.703632:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766737.703638:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703643:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703645:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703649:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703654:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766737.703662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766737.703667:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076594816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766737.703676:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703680:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775766737.703682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7ed80 x1862023076594176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766737.703683:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703692:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00100000:3.0:1775766737.703721:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076594176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766737.703722:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703733:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703744:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766737.703747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.703754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766737.703759:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076594944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766737.703760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766737.703780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766737.703782:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766737.703789:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766737.703791:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766737.703796:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3239 00000100:00080000:2.0:1775766737.703800:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766737.703804:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766737.703810:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766737.703816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766737.703829:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076595072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766737.703834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7ce00 x1862023076594304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.703863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076594304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.703905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.703911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.703916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766737.703935:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766737.703964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.703976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7df80 x1862023076594432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.703987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076594432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.703994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.703997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766737.704005:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766737.704020:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.704051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7ea00 x1862023076594560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.704062:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076594560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.704069:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.704072:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076594560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766737.704091:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766737.704107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.704115:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7d500 x1862023076594688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.704135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076594688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.704142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.704144:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704147:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766737.704151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766737.704167:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.704175:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7d880 x1862023076594816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.704185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076594816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.704191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.704193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704196:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766737.704200:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766737.704212:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.704218:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7fb80 x1862023076594944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.704227:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076594944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.704265:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.704269:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704272:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076594944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766737.704277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766737.704296:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766737.704305:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766737/real 1775766737] req@ffffa0434aa7c700 x1862023076595072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766737.704314:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076595072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766737.704321:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766737.704323:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766737.704326:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766737.895118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766737.895147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775766738.727149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766738.727166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766738.919135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766738.919135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766739.751130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766739.751236:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766739.943125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766739.943153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766740.775208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766740.775328:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766740.967116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766740.967116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766741.799092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766741.799127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766741.991098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766741.991098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766742.823151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766742.823157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766742.823160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766742.823166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3244 00010000:00010000:0.0:1775766742.823200:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766742.823219:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823224:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766742.823255:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076595200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823286:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766742.823288:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823292:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823297:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823301:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823310:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766742.823323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076595328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766742.823327:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076595200:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766742.823344:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823347:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766742.823349:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823354:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823359:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823363:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766742.823382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076595456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766742.823403:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823407:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3244 00000800:00000100:3.0:1775766742.823412:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766742.823413:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823419:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823424:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766742.823433:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076595584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823441:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823443:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766742.823444:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823447:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823449:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823452:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823455:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766742.823459:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7e300 x1862023076595200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766742.823463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076595712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823470:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823475:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766742.823478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823484:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823488:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823494:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823504:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766742.823511:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076595200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766742.823514:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076595840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823544:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823547:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766742.823547:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766742.823548:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823556:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823559:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766742.823559:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766742.823565:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823568:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076595200:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766742.823570:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766742.823580:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076595968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823585:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823588:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766742.823589:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823591:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823594:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823597:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595328:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766742.823601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766742.823610:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076596096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766742.823617:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766742.823620:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766742.823624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766742.823628:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3244 00000100:00080000:2.0:1775766742.823631:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766742.823634:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766742.823637:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766742.823648:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076596224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766742.823649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823673:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7f800 x1862023076595328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.823688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076595328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.823698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.823702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.823705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766742.823712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766742.823734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7f100 x1862023076595456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.823757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076595456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.823779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.823782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.823786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766742.823791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766742.823814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823825:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7c000 x1862023076595584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.823838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076595584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.823846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.823850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.823853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766742.823859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766742.823878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7dc00 x1862023076595712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.823900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076595712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.823909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.823912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.823922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076595712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766742.823929:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766742.823951:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.823980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7f480 x1862023076595840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.823993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076595840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.824002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.824005:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.824009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766742.824014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766742.824056:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.824068:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7e680 x1862023076595968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.824081:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076595968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.824089:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.824092:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.824096:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076595968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766742.824116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766742.824136:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.824155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7c380 x1862023076596096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.824168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076596096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.824191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.824196:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.824199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766742.824205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766742.824220:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766742.824227:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766742/real 1775766742] req@ffffa0434aa7e300 x1862023076596224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766742.824234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076596224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766742.824240:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766742.824242:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766742.824244:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766743.015133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766743.015133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766743.847123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766743.847162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766744.039095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766744.039140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766744.871147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766744.871269:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766745.063143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766745.063181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766745.895137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766745.895230:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766746.087141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766746.087179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766746.919186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766746.919245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766747.111097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766747.111120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766747.943112:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766747.943117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766747.943122:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766747.943141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775766747.943148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766747.943152:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943163:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943186:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766747.943222:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076596352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943247:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943251:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766747.943267:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943271:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943275:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766747.943277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076596352:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766747.943281:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943290:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766747.943305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076596480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766747.943316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943319:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943322:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943324:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766747.943334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076596608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943341:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766747.943344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943347:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943349:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943351:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943355:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766747.943356:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766747.943370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076596736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943376:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943378:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766747.943379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943382:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943385:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943388:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943392:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766747.943393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7c700 x1862023076596352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766747.943401:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076596864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943408:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766747.943412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076596352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766747.943413:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766747.943417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943426:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943429:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766747.943429:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766747.943436:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766747.943437:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766747.943443:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766747.943444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076596352:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766747.943456:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076596992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766747.943462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766747.943466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766747.943468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766747.943476:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943480:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943483:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943488:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766747.943489:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766747.943502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076597120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766747.943507:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7fb80 x1862023076596480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766747.943516:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766747.943523:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766747.943525:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766747.943532:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943534:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766747.943539:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076596480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766747.943542:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766747.943553:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076597248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766747.943558:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766747.943564:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766747.943566:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766747.943568:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766747.943572:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766747.943572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766747.943581:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3249 00000100:00080000:2.0:1775766747.943583:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766747.943587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766747.943588:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766747.943594:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766747.943604:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076597376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766747.943622:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943636:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7d880 x1862023076596608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943655:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076596608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943661:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943663:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943666:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766747.943670:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766747.943686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7d500 x1862023076596736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076596736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766747.943731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766747.943750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943758:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7ea00 x1862023076596864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076596864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943786:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943790:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943793:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076596864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766747.943800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766747.943822:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943833:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7df80 x1862023076596992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076596992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076596992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766747.943865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766747.943879:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7ce00 x1862023076597120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076597120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766747.943911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766747.943933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943940:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7ed80 x1862023076597248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076597248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766747.943960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766747.943970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766747.943975:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766747/real 1775766747] req@ffffa0434aa7c700 x1862023076597376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766747.943982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076597376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766747.943987:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766747.943989:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766747.943990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076597376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766748.135091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766748.135096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766748.967093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766748.967199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766749.159090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766749.159134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766749.991133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766749.991136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766750.183081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766750.183095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766751.015153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766751.015220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766751.207137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766751.207146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766752.039079:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766752.039193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766752.231175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766752.231175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766753.063151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766753.063156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766753.063164:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766753.063168:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063176:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3254 00010000:00010000:0.0:1775766753.063180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766753.063185:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063192:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766753.063224:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076597504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063247:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766753.063250:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766753.063251:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063254:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063257:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063260:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063268:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766753.063277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076597504:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766753.063288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076597632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063305:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766753.063307:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766753.063309:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063322:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766753.063334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766753.063343:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076597760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063354:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766753.063358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766753.063360:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063365:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063371:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063379:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7e300 x1862023076597504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766753.063383:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766753.063398:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076597888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766753.063407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076597504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766753.063411:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766753.063414:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063421:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:1.0:1775766753.063428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766753.063431:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766753.063435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766753.063439:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076597504:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766753.063444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766753.063455:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076598016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063464:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766753.063466:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597632:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766753.063471:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766753.063473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063476:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063479:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063482:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766753.063492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076598144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766753.063499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766753.063502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766753.063503:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063506:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063509:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063511:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063513:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766753.063520:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076598272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766753.063528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766753.063540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766753.063542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7c380 x1862023076597632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766753.063543:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766753.063546:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063554:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063560:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766753.063564:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076597632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766753.063566:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775766753.063576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766753.063586:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076598400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766753.063599:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597760:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766753.063600:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766753.063606:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766753.063608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766753.063612:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3254 00000100:00080000:2.0:1775766753.063616:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766753.063619:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775766753.063623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766753.063625:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766753.063636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076598528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766753.063639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7e680 x1862023076597760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076597760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766753.063691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766753.063722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7f480 x1862023076597888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076597888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063760:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076597888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766753.063773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766753.063787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7dc00 x1862023076598016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076598016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766753.063814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766753.063823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7c000 x1862023076598144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076598144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063860:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063864:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766753.063868:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766753.063883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7f100 x1862023076598272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076598272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063901:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.063905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766753.063909:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766753.063918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.063923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7f800 x1862023076598400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.063930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076598400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.063934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.063936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.065022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766753.065040:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766753.065060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766753.065069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766753/real 1775766753] req@ffffa0434aa7e300 x1862023076598528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766753.065095:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076598528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766753.065101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766753.065102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766753.065104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076598528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766753.255103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766753.255142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766754.087126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766754.087140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766754.279126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766754.279126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766755.111124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766755.111167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766755.303111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766755.303111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766756.135190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766756.135254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766756.327162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766756.327162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766757.159187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766757.159216:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766757.351144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766757.351144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766758.183125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766758.183146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775766758.183147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766758.183155:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766758.183160:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183170:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183180:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766758.183230:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766758.183268:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076598656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766758.183295:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183299:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766758.183301:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183305:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183310:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766758.183315:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183326:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766758.183336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076598656:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766758.183343:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076598784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766758.183356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766758.183361:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183366:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183371:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766758.183375:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183381:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766758.183400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766758.183412:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076598912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766758.183423:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766758.183429:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183433:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183439:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766758.183443:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349643100 x1862023076598656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766758.183447:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183457:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766758.183470:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076599040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766758.183477:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076598656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766758.183489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766758.183493:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766758.183494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766758.183498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766758.183498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183505:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076598656:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766758.183509:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183514:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766758.183522:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183527:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766758.183541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766758.183543:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076599168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766758.183558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183562:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766758.183564:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766758.183567:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.183578:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349641500 x1862023076598784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766758.183582:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183592:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766758.183596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076598784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766758.183601:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766758.183610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766758.183612:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766758.183615:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766758.183624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766758.183631:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076599296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766758.183643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775766758.183644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766758.183650:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766758.183652:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183659:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00100000:1.0:1775766758.183662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349642a00 x1862023076598912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766758.183664:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766758.183673:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183683:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766758.183687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076598912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766758.183699:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076599424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766758.183700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.183707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183714:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076598912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766758.183726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599040:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766758.183727:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766758.183735:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183741:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183746:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766758.183749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766758.183756:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766758.183761:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349643800 x1862023076599040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766758.183762:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766758.183777:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076599552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766758.183780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076599040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.183793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.183800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183807:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599040:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766758.183811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766758.183815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766758.183817:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766758.183823:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3259 00000100:00080000:0.0:1775766758.183828:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766758.183832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599168:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766758.183833:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766758.183843:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766758.183857:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076599680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766758.183867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.183881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349643b80 x1862023076599168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766758.183891:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076599168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.183898:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.183900:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183902:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766758.183906:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766758.183918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.183924:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349643100 x1862023076599296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766758.183931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076599296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.183946:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.183949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183951:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766758.183955:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766758.183967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.183974:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349641880 x1862023076599424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766758.183981:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076599424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.183986:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.183988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.183990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766758.183993:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766758.184002:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.184007:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349640380 x1862023076599552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766758.184014:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076599552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.184072:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.184075:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.184082:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766758.184096:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766758.184112:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766758.184119:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766758/real 1775766758] req@ffffa04349642300 x1862023076599680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766758.184127:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076599680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766758.184132:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766758.184134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766758.184136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076599680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766758.375133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766758.375134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766759.207173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766759.207233:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766759.399120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766759.399175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766760.231127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766760.231216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766760.423116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766760.423117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766761.255085:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766761.255151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766761.447135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766761.447139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766762.279122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766762.279162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766762.471137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766762.471137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766763.303172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766763.303185:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303191:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766763.303215:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775766763.303225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766763.303227:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00080000:2.0:1775766763.303240:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303246:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766763.303276:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076599808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766763.303298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303302:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766763.303305:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303309:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00080000:2.0:1775766763.303314:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303318:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766763.303328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766763.303334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076599808:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766763.303340:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076599936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766763.303350:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303353:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766763.303355:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303359:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00080000:2.0:1775766763.303362:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303365:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766763.303370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766763.303378:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076600064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766763.303389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766763.303408:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303412:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766763.303413:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303416:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00080000:2.0:1775766763.303420:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303424:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303425:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7c700 x1862023076599808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766763.303429:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766763.303440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076600192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766763.303449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303452:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766763.303454:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766763.303456:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076599808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766763.303458:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00080000:2.0:1775766763.303461:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303465:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775766763.303467:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766763.303469:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775766763.303470:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303475:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076599808:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766763.303477:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076600320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766763.303485:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303487:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766763.303489:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303492:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00100000:1.0:1775766763.303495:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599936:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766763.303500:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303505:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766763.303509:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766763.303516:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766763.303519:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076600448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766763.303528:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7ed80 x1862023076599936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766763.303529:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766763.303534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303538:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00100000:1.0:1775766763.303540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076599936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766763.303542:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303545:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766763.303549:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775766763.303550:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303553:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303556:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076599936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766763.303558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076600576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766763.303562:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600064:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766763.303567:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766763.303572:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766763.303576:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3264 00000800:00000100:1.0:1775766763.303580:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766763.303580:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303583:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766763.303587:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766763.303589:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7ce00 x1862023076600064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766763.303596:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076600704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766763.303601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076600064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766763.303616:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766763.303619:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766763.303620:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766763.303621:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766763.303623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766763.303624:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3264 00000100:00100000:1.0:1775766763.303627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600064:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766763.303628:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766763.303632:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600192:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775766763.303637:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766763.303647:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076600832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766763.303652:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7df80 x1862023076600192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076600192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303683:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303689:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766763.303695:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766763.303711:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7ea00 x1862023076600320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303726:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076600320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076600320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766763.303755:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766763.303768:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7d500 x1862023076600448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303782:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076600448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766763.303794:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775766763.303803:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7d880 x1862023076600576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076600576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766763.303826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766763.303835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7fb80 x1862023076600704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076600704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766763.303876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766763.303887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766763.303892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766763/real 1775766763] req@ffffa0434aa7c700 x1862023076600832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766763.303899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076600832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766763.303904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766763.303906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775766763.303907:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076600832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766763.495120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766763.495131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766764.327138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766764.327194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766764.519117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766764.519118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766765.351113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766765.351145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766765.543100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766765.543101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766766.375146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766766.375227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766766.567103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766766.567104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775766767.399139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766767.399153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766767.591103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766767.591103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766768.423174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766768.423183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766768.423199:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423212:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3270 00010000:00010000:3.0:1775766768.423224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766768.423250:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423257:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766768.423313:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076600960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766768.423357:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423361:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423366:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423370:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423381:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766768.423391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076600960:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766768.423395:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076601088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766768.423408:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423422:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423425:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423428:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766768.423438:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076601216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423445:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766768.423455:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423458:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423461:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423464:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766768.423479:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076601344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423490:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766768.423496:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423500:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423517:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423523:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423530:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766768.423536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766768.423547:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076601472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766768.423567:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766768.423577:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766768.423578:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423586:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423591:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766768.423607:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766768.423616:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076601600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766768.423630:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601472:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766768.423631:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423655:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423661:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423664:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423669:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766768.423675:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766768.423682:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076601728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766768.423695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423701:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766768.423702:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423705:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423707:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766768.423710:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423714:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766768.423724:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076601856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766768.423746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766768.423749:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766768.423753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766768.423760:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766768.423765:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3270 00000100:00080000:1.0:1775766768.423770:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766768.423773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601856:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775766768.423776:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766768.423783:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766768.423795:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076601984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766768.423835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766768.615118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766768.615162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766769.447115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766769.447209:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766769.639093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766769.639099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766770.471146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766770.471149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766770.663115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766770.663116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766771.495175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766771.495272:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775766771.495399:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775766771.495435:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775766771.495449:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495504:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495518:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495528:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495537:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495550:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495564:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775766771.495573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350930e00 x1862023076601984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775766771.495579:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775766771.495603:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775766771.495637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076601984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495667:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766771.495687:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350930000 x1862023076601856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495704:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076601856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766771.495754:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350930a80 x1862023076601728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076601728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766771.495796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350931880 x1862023076601472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495811:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076601472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495823:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495828:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766771.495840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350932680 x1862023076601600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076601600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495864:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495874:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076601600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766771.495908:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350932a00 x1862023076601088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076601088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766771.495954:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350931c00 x1862023076601216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.495966:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076601216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.495975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.495979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.495982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766771.495992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350930700 x1862023076601344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.496004:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076601344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.496012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.496017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.496020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076601344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766771.496070:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766768/real 1775766771] req@ffffa04350932300 x1862023076600960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766771.496087:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076600960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766771.496096:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766771.496100:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766771.496105:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076600960:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766771.687110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766771.687111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766772.519208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766772.519217:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766772.711103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766772.711105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766773.543149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766773.543175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766773.543245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766773.543258:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543270:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543281:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543288:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766773.543330:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076602112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766773.543374:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543377:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543380:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543383:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543395:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766773.543405:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076602240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766773.543417:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543420:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543422:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543424:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766773.543432:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076602112:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766773.543450:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766773.543463:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076602368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766773.543479:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543484:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543489:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543503:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766773.543519:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076602496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543530:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543534:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766773.543536:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543541:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543546:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543550:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543557:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775766773.543560:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766773.543574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076602624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766773.543598:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543604:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543609:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543615:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543623:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766773.543642:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076602752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766773.543650:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349641180 x1862023076602112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766773.543657:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766773.543670:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543680:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543686:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543691:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766773.543695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076602112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766773.543700:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766773.543719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766773.543722:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076602880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766773.543731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766773.543742:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766773.543742:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076602112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766773.543751:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766773.543754:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543759:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543763:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543766:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543773:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766773.543794:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766773.543799:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076603008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766773.543812:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766773.543817:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766773.543819:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766773.543824:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3275 00000100:00080000:0.0:1775766773.543830:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766773.543837:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766773.543843:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766773.543848:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766773.543881:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076603136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766773.543908:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349641c00 x1862023076602240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.543935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076602240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.543947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.544642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.544653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766773.544664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766773.544716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.544741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349640700 x1862023076602368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.544757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076602368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.544767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.544771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.544775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766773.544782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766773.544808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.544821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349642680 x1862023076602496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.544843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076602496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.544881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.544885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.544889:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766773.544897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766773.544923:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.544935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349640a80 x1862023076602624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.544948:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076602624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.544958:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.544962:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.544965:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076602624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766773.544972:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076602752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766773.544991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.544999:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349640e00 x1862023076602752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.545007:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076602752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.545013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.545016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.545043:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076602752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766773.545048:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076602880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766773.545065:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.545073:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349642300 x1862023076602880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.545081:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076602880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.545086:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.545088:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.545091:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076602880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766773.545116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766773.545126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.545133:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349640380 x1862023076603008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.545146:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076603008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.545151:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.545153:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.545156:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766773.545159:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766773.545192:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766773.545212:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766773/real 1775766773] req@ffffa04349641880 x1862023076603136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766773.545226:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076603136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766773.545235:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766773.545238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766773.545240:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766773.735103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766773.735103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766774.567153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766774.567218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766774.759121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766774.759121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766775.591199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766775.591268:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766775.783099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766775.783143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766776.615086:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766776.615196:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766776.807088:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766776.807092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766777.639128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766777.639204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766777.831098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766777.831119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766778.663127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766778.663142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766778.663160:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775766778.663163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766778.663172:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663183:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663191:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766778.663230:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076603264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766778.663282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663286:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766778.663288:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663293:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663298:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663303:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663313:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766778.663328:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076603392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766778.663337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076603264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766778.663347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766778.663353:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663359:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663363:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663367:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663373:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766778.663385:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076603520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766778.663395:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766778.663401:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766778.663405:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766778.663410:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663421:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663426:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663433:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766778.663449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349643100 x1862023076603264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766778.663450:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076603648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766778.663476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766778.663481:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663487:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00100000:3.0:1775766778.663489:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076603264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766778.663492:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663501:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766778.663501:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.663510:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766778.663511:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766778.663514:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076603264:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766778.663528:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076603776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766778.663536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603392:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766778.663541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766778.663552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663557:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663563:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766778.663569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766778.663570:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663577:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766778.663584:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349643b80 x1862023076603392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766778.663592:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076603904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766778.663615:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076603392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.663635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.663641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766778.663645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766778.663645:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603392:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766778.663654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766778.663656:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766778.663658:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663665:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00080000:0.0:1775766778.663680:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766778.663685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766778.663686:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663695:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766778.663697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349643800 x1862023076603520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.663707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076603520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.663713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766778.663714:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076604032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766778.663717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.663725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766778.663735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603648:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766778.663752:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663756:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766778.663760:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766778.663760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766778.663767:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00100000:3.0:1775766778.663769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349642a00 x1862023076603648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766778.663774:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766778.663781:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663792:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766778.663797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076603648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766778.663823:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076604160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766778.663827:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.663833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.663840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603648:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766778.663845:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766778.663850:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766778.663852:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766778.663858:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3280 00000100:00100000:3.0:1775766778.663859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603776:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766778.663874:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766778.663878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766778.663882:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766778.663890:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766778.663890:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349641500 x1862023076603776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.663903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076603776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.663910:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766778.663912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076604288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766778.663915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.663921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076603776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766778.663932:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766778.663949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766778.663956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349643100 x1862023076603904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.663968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076603904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.663989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.663993:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.663997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076603904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766778.664003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766778.664044:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766778.664059:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349641880 x1862023076604032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.664068:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076604032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.664073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.664076:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.664078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766778.664082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766778.664095:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766778.664100:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349640380 x1862023076604160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.664107:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076604160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.664112:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.664114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.664120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766778.664134:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766778.664147:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766778.664153:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766778/real 1775766778] req@ffffa04349642300 x1862023076604288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766778.664161:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076604288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766778.664166:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766778.664168:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766778.664169:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076604288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766778.855118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766778.855119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766779.687168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766779.687205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766779.879131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766779.879132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766780.711162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766780.711194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766780.903172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766780.903196:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766781.735127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766781.735173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766781.927193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766781.927194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766782.759141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766782.759169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775766782.951092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766782.951229:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766783.783174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766783.783241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766783.783250:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.783260:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783276:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783284:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766783.783327:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076604416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775766783.783337:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766783.783355:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766783.783361:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.783364:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783370:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783377:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.783417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766783.783429:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076604416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766783.783441:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076604544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766783.783458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766783.783466:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.783505:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783520:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783526:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.783534:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766783.783552:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766783.783574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076604672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766783.783586:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766783.783591:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.783596:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783602:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783607:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.783614:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766783.783615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7e300 x1862023076604416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766783.783653:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076604800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766783.783653:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076604416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.783689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766783.783695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783699:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766783.783701:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766783.783717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766783.783719:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783727:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766783.783727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076604416:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766783.783741:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.783748:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766783.783783:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076604928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766783.783787:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604544:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766783.783814:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775766783.783820:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766783.783828:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766783.783832:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766783.783837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7f800 x1862023076604544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766783.783843:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783858:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783872:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766783.783874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076604544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766783.783888:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766783.783901:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.783907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766783.783913:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076605056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766783.783917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766783.783943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766783.783944:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.783963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766783.783966:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.783971:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.783978:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.783982:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766783.783988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766783.783994:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766783.784022:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076605184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766783.784031:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7f100 x1862023076604672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766783.784058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.784060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766783.784062:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766783.784066:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.784077:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.784086:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.784095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766783.784100:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076604672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766783.784114:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076605312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766783.784115:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784132:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784151:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766783.784160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766783.784165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766783.784166:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766783.784171:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604800:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766783.784173:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3285 00000100:00080000:2.0:1775766783.784176:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766783.784179:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766783.784192:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766783.784215:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076605440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766783.784241:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784256:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7c000 x1862023076604800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076604800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784273:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784276:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784279:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766783.784296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766783.784309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784316:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7dc00 x1862023076604928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784325:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076604928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784332:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784335:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784337:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076604928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766783.784342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766783.784354:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784360:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7f480 x1862023076605056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784388:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076605056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784399:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784403:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784407:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766783.784415:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766783.784439:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7e680 x1862023076605184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076605184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784466:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784468:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784471:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766783.784475:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766783.784510:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784520:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7c380 x1862023076605312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784532:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076605312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784557:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784561:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766783.784571:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766783.784593:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766783.784603:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766783/real 1775766783] req@ffffa0434aa7e300 x1862023076605440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766783.784615:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076605440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766783.784624:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766783.784627:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766783.784630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076605440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766783.975137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766783.975138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766784.807121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766784.807207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766784.999128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766784.999137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766785.831110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766785.831257:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766786.023168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766786.023168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766786.855124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766786.855166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766787.047136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766787.047165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766787.879135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766787.879214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766788.071134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766788.071135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766788.903159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766788.903187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766788.903197:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903206:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903217:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903224:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775766788.903238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766788.903259:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076605568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766788.903304:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903309:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903315:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903319:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903329:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766788.903342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076605568:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766788.903344:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076605696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766788.903362:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903366:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903369:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903371:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766788.903386:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076605824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766788.903398:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903400:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903410:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903413:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766788.903418:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903420:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766788.903432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076605952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766788.903444:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903447:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903450:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903453:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766788.903463:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076606080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903471:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766788.903472:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903475:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903477:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903479:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903482:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766788.903490:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076606208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766788.903521:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766788.903526:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349640e00 x1862023076605568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766788.903528:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903533:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903539:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766788.903561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076605568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766788.903563:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076606336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766788.903578:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903582:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903586:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903590:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903596:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766788.903608:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076606464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766788.903620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766788.903624:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766788.903626:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766788.903631:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3290 00000100:00080000:0.0:1775766788.903643:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766788.903648:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766788.903661:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766788.903673:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076606592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766788.903678:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.903686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.903691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076605568:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766788.903789:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766788.903824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.903852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349640a80 x1862023076605696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.903863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076605696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.903870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.903873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.903875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766788.903880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766788.903893:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.903900:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349642680 x1862023076605824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.903908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076605824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.903924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.903927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.903929:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766788.903934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766788.903949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.903956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349640700 x1862023076605952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.903964:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076605952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.903970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.903972:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.903974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076605952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766788.903978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766788.903988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.903994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349641c00 x1862023076606080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.904002:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076606080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.904008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.904010:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.904019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766788.904027:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766788.904085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.904121:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349641180 x1862023076606208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.904135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076606208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.904146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.904150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.904155:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766788.904161:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766788.904194:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.904201:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349642300 x1862023076606336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.904210:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076606336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.904215:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.904217:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.904219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766788.904224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766788.904234:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.904250:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349640380 x1862023076606464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.904259:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076606464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.904265:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.904267:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.904269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766788.904273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766788.904285:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766788.904291:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766788/real 1775766788] req@ffffa04349641880 x1862023076606592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766788.904299:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076606592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766788.904305:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766788.904307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766788.904309:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076606592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766789.095149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766789.095149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766789.927130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766789.927304:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766790.119170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766790.119170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766790.951126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766790.951156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766791.143146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766791.143162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766791.975090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766791.975206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766792.167115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766792.167132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766792.999106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766792.999200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766793.191140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766793.191238:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766794.023141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1775766794.023160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766794.023165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766794.023170:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023179:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023187:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023193:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766794.023233:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076606720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023253:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023256:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766794.023257:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023260:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023263:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766794.023281:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076606848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023288:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023291:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766794.023292:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023295:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023297:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023301:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023302:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076606720:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766794.023304:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766794.023311:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076606976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766794.023347:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023350:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023355:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023358:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023362:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766794.023368:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766794.023372:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076607104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766794.023383:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023386:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023388:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023390:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023393:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766794.023401:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076607232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766794.023408:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7c700 x1862023076606720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766794.023409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766794.023412:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023420:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023428:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766794.023431:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076606720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766794.023433:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023440:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766794.023446:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766794.023452:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076607360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766794.023453:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023461:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076606720:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766794.023469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766794.023473:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766794.023475:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023482:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023483:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023486:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023489:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766794.023497:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766794.023501:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076607488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766794.023508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766794.023510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7fb80 x1862023076606848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766794.023511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766794.023514:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023521:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023526:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023531:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766794.023534:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766794.023534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076606848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766794.023545:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076607616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766794.023545:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023559:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766794.023571:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766794.023574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766794.023575:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766794.023578:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3295 00000100:00080000:2.0:1775766794.023581:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766794.023586:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023586:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606976:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766794.023594:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766794.023606:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766794.023612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076607744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766794.023617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7d880 x1862023076606976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076606976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023672:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023674:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023676:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076606976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766794.023681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076607104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766794.023696:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7d500 x1862023076607104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076607104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023720:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076607104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766794.023723:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076607232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766794.023732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023738:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7ea00 x1862023076607232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076607232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076607232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766794.023775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766794.023789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7df80 x1862023076607360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076607360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766794.023816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766794.023825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023830:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7ce00 x1862023076607488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076607488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023843:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766794.023853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766794.023865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023871:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7ed80 x1862023076607616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076607616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766794.023899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766794.023909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766794.023915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766794/real 1775766794] req@ffffa0434aa7fb80 x1862023076607744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766794.023922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076607744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766794.023927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766794.023929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766794.023930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076607744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766794.215089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766794.215091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766795.047146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766795.047207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766795.239147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766795.239148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766796.071116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766796.071159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766796.263134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766796.263205:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766797.095179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766797.095241:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766797.287150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766797.287161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766798.119178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766798.119280:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766798.311154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766798.311181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766799.143211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766799.143219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766799.143221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143231:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766799.143236:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143248:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.143261:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143270:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766799.143326:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076607872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766799.143386:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143393:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.143399:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143405:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766799.143419:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076607872:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766799.143439:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076608000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143454:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766799.143461:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143466:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.143472:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143475:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143480:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766799.143493:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076608128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143506:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766799.143508:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143511:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3300 00000800:00000100:3.0:1775766799.143518:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766799.143521:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143528:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143543:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766799.143559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076608256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766799.143580:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349643100 x1862023076607872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766799.143581:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143620:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.143631:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143638:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143644:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766799.143653:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076607872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766799.143661:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076608384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766799.143697:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766799.143706:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766799.143710:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766799.143711:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766799.143725:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00100000:3.0:1775766799.143725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076607872:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766799.143736:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143742:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143747:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766799.143766:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766799.143772:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076608512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766799.143801:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143807:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3300 00000800:00000100:3.0:1775766799.143814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766799.143819:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143827:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.143835:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766799.143838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349641500 x1862023076608000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766799.143863:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076608640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766799.143869:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076608000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.143898:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.143907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766799.143910:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766799.143915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608000:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766799.143917:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766799.143920:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.143926:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.143935:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766799.143941:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766799.143947:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608128:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766799.143949:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766799.143968:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076608768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766799.143979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766799.143986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775766799.143987:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766799.143992:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766799.144000:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3300 00000100:00080000:0.0:1775766799.144005:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766799.144010:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349642a00 x1862023076608128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766799.144014:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766799.144069:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766799.144076:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076608128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144103:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144108:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766799.144117:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076608896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766799.144120:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766799.144165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144179:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349643800 x1862023076608256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144193:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076608256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144202:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144205:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144208:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766799.144214:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766799.144229:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144238:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349643b80 x1862023076608384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076608384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144270:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144273:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144276:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076608384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766799.144298:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766799.144326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349643100 x1862023076608512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144349:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076608512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144358:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144360:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144363:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766799.144369:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766799.144384:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144392:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349641880 x1862023076608640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144404:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076608640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144427:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144431:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144434:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766799.144441:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766799.144461:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349640380 x1862023076608768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144483:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076608768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144491:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144493:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144496:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766799.144502:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766799.144517:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766799.144525:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766799/real 1775766799] req@ffffa04349642300 x1862023076608896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766799.144537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076608896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766799.144545:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766799.144550:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766799.144564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076608896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766799.335164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766799.335164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766800.167152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766800.167304:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766800.359155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766800.359188:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766801.191172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766801.191249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766801.383120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766801.383120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766802.215089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766802.215161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766802.407126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766802.407207:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766803.239154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766803.239225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766803.431160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766803.431168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766804.263162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766804.263220:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263232:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766804.263238:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.263262:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263278:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766804.263288:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766804.263340:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766804.263352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076609024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766804.263392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263398:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766804.263400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.263405:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263411:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766804.263418:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.263431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766804.263463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076609152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766804.263464:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076609024:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766804.263525:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263530:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766804.263533:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.263571:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263583:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766804.263588:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766804.263599:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.263613:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766804.263640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076609280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766804.263650:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349641180 x1862023076609024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766804.263662:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263678:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766804.263684:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.263691:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263697:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766804.263699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076609024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766804.263707:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.263715:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766804.263733:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076609408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766804.263734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.263750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.263765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076609024:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766804.263768:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263778:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766804.263781:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.263787:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263803:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766804.263806:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766804.263816:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.263827:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766804.263849:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076609536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766804.263859:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766804.263880:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263884:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766804.263887:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766804.263891:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349641c00 x1862023076609152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766804.263898:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.263910:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766804.263924:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.263930:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766804.263931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076609152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.263951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766804.263954:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076609664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766804.263967:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.263973:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766804.263978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766804.263979:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766804.263990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766804.263992:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.264005:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766804.264013:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766804.264015:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.264063:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766804.264079:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076609792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766804.264116:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766804.264121:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766804.264125:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766804.264130:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766804.264137:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00080000:0.0:1775766804.264149:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766804.264155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349640700 x1862023076609280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766804.264159:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.264175:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766804.264187:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076609280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766804.264199:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076609920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766804.264206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264224:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766804.264230:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766804.264237:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766804.264239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766804.264246:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766804.264256:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3305 00000100:00100000:3.0:1775766804.264264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609408:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766804.264269:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766804.264277:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766804.264285:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766804.264302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766804.264307:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076610048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766804.264323:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349642680 x1862023076609408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264345:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076609408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264357:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264362:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264366:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766804.264402:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766804.264441:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766804.264467:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349640a80 x1862023076609536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264492:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076609536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264509:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264520:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076609536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766804.264532:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766804.264600:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766804.264623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349640e00 x1862023076609664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076609664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264665:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264670:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766804.264682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766804.264710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766804.264724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349641180 x1862023076609792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076609792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766804.264783:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766804.264807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766804.264819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349642300 x1862023076609920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076609920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264848:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076609920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766804.264854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766804.264871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766804.264881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766804/real 1775766804] req@ffffa04349640380 x1862023076610048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766804.264894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076610048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766804.264919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766804.264923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766804.264927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766804.455194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766804.455194:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766805.287131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766805.287156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766805.479114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766805.479201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766806.311139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766806.311216:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766806.503137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766806.503176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766807.335177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766807.335189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766807.527119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766807.527120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766808.359144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766808.359231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766808.551154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766808.551166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766809.383159:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775766809.383164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766809.383167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766809.383175:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766809.383182:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383205:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383212:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766809.383259:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076610176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766809.383288:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383291:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766809.383292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383295:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383298:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383301:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383312:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766809.383323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076610304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766809.383332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076610176:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766809.383335:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766809.383353:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383356:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383359:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383361:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383388:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766809.383405:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076610432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766809.383420:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766809.383423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383428:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766809.383430:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383434:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383439:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383444:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383451:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766809.383464:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076610560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766809.383475:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349641880 x1862023076610176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766809.383477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383500:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766809.383501:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383507:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383519:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383528:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766809.383539:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076610176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766809.383547:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076610688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766809.383569:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766809.383572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766809.383573:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766809.383579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766809.383581:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766809.383587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076610176:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766809.383591:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383602:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383610:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766809.383624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766809.383626:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076610816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766809.383644:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383647:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766809.383650:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383654:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383657:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383661:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766809.383662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383675:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766809.383688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349643100 x1862023076610304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766809.383689:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076610944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766809.383696:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766809.383698:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766809.383699:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383705:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383713:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766809.383724:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383732:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766809.383744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076610304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766809.383752:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076611072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766809.383756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766809.383770:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766809.383770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766809.383777:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766809.383781:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766809.383782:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766809.383795:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3311 00000100:00080000:0.0:1775766809.383799:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766809.383800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766809.383807:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766809.383816:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766809.383831:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076611200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766809.383843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.383857:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349643b80 x1862023076610432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.383870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076610432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.383877:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.383881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.383884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766809.383891:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766809.383911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.383922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349643800 x1862023076610560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.383935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076610560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.383955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.383958:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.383960:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766809.383964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766809.383980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.383988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349642a00 x1862023076610688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.383995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076610688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.384001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.384003:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.384005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076610688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766809.384021:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766809.384060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.384071:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349641500 x1862023076610816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.384084:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076610816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.384094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.384097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.384107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766809.384112:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766809.384128:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.384135:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349641880 x1862023076610944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.384143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076610944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.384148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.384150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.384152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076610944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766809.384156:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766809.384165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.384171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349640380 x1862023076611072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.384177:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076611072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.384182:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.384184:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.384186:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766809.384189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766809.384198:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766809.384210:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766809/real 1775766809] req@ffffa04349642300 x1862023076611200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766809.384218:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076611200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766809.384223:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766809.384225:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766809.384227:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766809.575081:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766809.575178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766810.407353:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766810.407383:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766810.599172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766810.599172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766811.431159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766811.431226:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766811.623129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766811.623137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766812.455141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766812.455175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766812.647123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766812.647126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766813.479112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766813.479127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766813.671131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766813.671147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766814.503227:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766814.503247:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766814.503264:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766814.503269:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766814.503272:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503289:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503322:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766814.503375:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076611328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766814.503410:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503415:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766814.503417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.503422:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503450:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503459:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766814.503480:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766814.503497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076611328:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766814.503504:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076611456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766814.503523:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503528:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766814.503530:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.503535:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503540:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503545:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766814.503552:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766814.503565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076611584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766814.503578:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775766814.503602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766814.503603:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.503615:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503626:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503635:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766814.503646:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766814.503666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349641180 x1862023076611328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766814.503678:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076611712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766814.503711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076611328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766814.503713:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503728:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766814.503734:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.503750:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:3.0:1775766814.503750:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766814.503761:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766814.503767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766814.503773:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766814.503780:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076611328:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766814.503794:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766814.503814:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076611840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766814.503828:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766814.503830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611456:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766814.503833:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766814.503839:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.503847:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503851:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503854:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766814.503861:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766814.503894:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076611968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766814.503912:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.503918:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766814.503921:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766814.503930:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766814.503935:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.503947:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.503956:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766814.503964:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349640e00 x1862023076611456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766814.503970:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766814.503993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076611456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766814.504003:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076612096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766814.504015:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504096:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611456:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766814.504100:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766814.504109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766814.504112:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766814.504121:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766814.504133:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.504147:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.504158:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766814.504160:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766814.504176:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766814.504185:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349640a80 x1862023076611584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504211:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076611584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766814.504213:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076612224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766814.504259:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504269:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766814.504270:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766814.504280:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766814.504281:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766814.504285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766814.504294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611712:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766814.504311:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3316 00000100:00080000:0.0:1775766814.504320:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766814.504327:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766814.504336:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766814.504337:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349642680 x1862023076611712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766814.504373:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076612352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766814.504418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076611712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504456:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504488:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766814.504519:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766814.504546:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504560:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349640700 x1862023076611840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504573:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076611840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504583:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504590:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076611840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766814.504597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766814.504614:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349641c00 x1862023076611968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504636:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076611968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076611968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766814.504678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766814.504709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504720:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349641180 x1862023076612096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076612096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504742:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504745:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504748:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766814.504754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766814.504771:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349642300 x1862023076612224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076612224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766814.504851:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766814.504891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766814.504913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766814/real 1775766814] req@ffffa04349640380 x1862023076612352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766814.504932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076612352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766814.504941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766814.504945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766814.504948:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076612352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766814.695149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766814.695149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766815.527155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766815.527305:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766815.719159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766815.719159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766816.551135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766816.551185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766816.743107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766816.743107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766817.575123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766817.575218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766817.767132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766817.767142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766818.599129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766818.599233:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766818.791186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766818.791188:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766819.623120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766819.623120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775766819.623131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766819.623133:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623142:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623151:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623157:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766819.623189:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076612480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766819.623214:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623217:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623220:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623223:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623240:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076612480:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775766819.623244:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766819.623256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076612608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766819.623269:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623272:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623275:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623277:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766819.623281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766819.623287:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076612736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775766819.623295:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766819.623300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766819.623301:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623305:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623308:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623311:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766819.623315:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766819.623322:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076612864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766819.623333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350932300 x1862023076612480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766819.623338:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623347:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623351:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623354:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076612480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766819.623359:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775766819.623370:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766819.623372:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076612992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766819.623396:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766819.623401:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766819.623404:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076612480:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766819.623407:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623424:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623427:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766819.623431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766819.623442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076613120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766819.623442:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766819.623455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766819.623459:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623462:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623465:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623468:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766819.623471:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766819.623472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623485:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350930700 x1862023076612608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766819.623491:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076613248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766819.623501:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076612608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766819.623502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766819.623511:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766819.623517:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:3.0:1775766819.623518:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766819.623520:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766819.623524:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766819.623527:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623531:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612608:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775766819.623532:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766819.623540:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766819.623545:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076613376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766819.623552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766819.623554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766819.623556:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766819.623560:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766819.623563:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3321 00000100:00080000:1.0:1775766819.623567:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766819.623571:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623574:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350931c00 x1862023076612736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766819.623575:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766819.623589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076613504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766819.623590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076612736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623616:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766819.623627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766819.623649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350932a00 x1862023076612864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623664:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076612864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623669:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623672:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766819.623677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766819.623686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350932680 x1862023076612992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623698:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076612992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623707:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076612992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766819.623718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766819.623727:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350931880 x1862023076613120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076613120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766819.623763:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766819.623777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623787:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350930a80 x1862023076613248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623799:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076613248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766819.623831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766819.623850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623859:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350930000 x1862023076613376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076613376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623880:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623882:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766819.623888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766819.623901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766819.623907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766819/real 1775766819] req@ffffa04350930e00 x1862023076613504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766819.623914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076613504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766819.623919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766819.623921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766819.623922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076613504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766819.815096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766819.815096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766820.647105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766820.647179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766820.839099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766820.839102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766821.671088:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766821.671129:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766821.863111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766821.863111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766822.695144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766822.695189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766822.887149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766822.887149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766823.719111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766823.719137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766823.911101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766823.911150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766824.743106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766824.743143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766824.743155:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775766824.743158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766824.743178:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743189:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743198:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766824.743230:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076613632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766824.743260:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743265:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766824.743271:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076613632:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766824.743274:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743283:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766824.743297:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076613760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766824.743315:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743319:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743340:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743344:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766824.743348:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743350:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766824.743362:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076613888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766824.743391:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743395:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743400:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743405:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743411:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766824.743422:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076614016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766824.743435:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743439:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743442:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743446:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743450:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766824.743459:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076614144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766824.743474:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743477:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743488:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743492:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743498:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766824.743508:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076614272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766824.743522:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743526:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743530:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743533:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743538:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766824.743547:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076614400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743560:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775766824.743562:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350930e00 x1862023076613632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766824.743563:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743567:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743571:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743575:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766824.743581:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766824.743591:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076614528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766824.743613:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766824.743617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766824.743620:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766824.743625:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3326 00000100:00080000:1.0:1775766824.743629:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766824.743634:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076613632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766824.743640:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766824.743648:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766824.743652:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076614656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766824.743653:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743658:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076613632:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766824.743680:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076613760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766824.743704:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350930000 x1862023076613760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743724:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076613760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743731:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076613760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766824.743740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076613888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766824.743752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743759:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350930a80 x1862023076613888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076613888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076613888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766824.743799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766824.743814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350931880 x1862023076614016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076614016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743836:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766824.743841:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766824.743850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350932680 x1862023076614144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076614144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743867:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743869:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766824.743879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766824.743892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743906:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350932a00 x1862023076614272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076614272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766824.743926:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766824.743935:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743940:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350931c00 x1862023076614400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076614400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.743953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.743955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.743956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766824.743959:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766824.743968:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.743981:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350930700 x1862023076614528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.743989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076614528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.744005:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.744007:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.744009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766824.744012:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766824.744023:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766824.744049:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766824/real 1775766824] req@ffffa04350932300 x1862023076614656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766824.744062:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076614656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766824.744071:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766824.744074:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766824.744078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076614656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766824.935138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766824.935178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766825.767195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766825.767225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766825.959110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766825.959157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766826.791131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766826.791247:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766826.983096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766826.983115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766827.815106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766827.815130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766828.007122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766828.007124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766828.839099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766828.839194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766829.031095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766829.031110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766829.863128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766829.863130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766829.863141:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766829.863147:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863159:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863170:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775766829.863176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766829.863187:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766829.863225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076614784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766829.863280:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863285:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863290:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863295:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863322:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766829.863336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076614784:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766829.863340:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076614912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766829.863361:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863365:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863369:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863372:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863376:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766829.863388:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076615040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766829.863401:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863405:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863416:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863422:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766829.863430:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863432:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766829.863455:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076615168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766829.863485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863488:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863492:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863500:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766829.863513:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7c700 x1862023076614784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766829.863514:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076615296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863551:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766829.863552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863556:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863559:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863563:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766829.863568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076614784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766829.863569:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766829.863582:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076615424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766829.863593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.863602:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.863613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076614784:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766829.863615:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766829.863624:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863629:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863633:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863636:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863640:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766829.863649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766829.863653:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076615552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766829.863664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766829.863669:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863672:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00080000:2.0:1775766829.863683:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863689:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766829.863692:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766829.863696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766829.863708:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076615680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766829.863718:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7e300 x1862023076614912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766829.863720:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766829.863729:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766829.863733:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766829.863745:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3331 00000100:00100000:3.0:1775766829.863751:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076614912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766829.863761:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766829.863773:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766829.863773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.863784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766829.863786:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766829.863792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076614912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766829.863804:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766829.863809:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076615808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766829.863842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.863859:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7c380 x1862023076615040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.863894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076615040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.863905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.863909:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.863913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766829.863922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766829.863950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.863964:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7e680 x1862023076615168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.863980:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076615168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.863990:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.863994:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.863998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766829.864006:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766829.864060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.864075:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7f480 x1862023076615296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.864092:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076615296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.864121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.864126:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.864131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076615296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766829.864139:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766829.864169:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.864183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7dc00 x1862023076615424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.864199:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076615424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.864211:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.864215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.864219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766829.864244:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766829.864286:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.864301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7c000 x1862023076615552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.864335:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076615552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.864346:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.864350:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.864364:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766829.864373:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766829.864404:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.864419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7f100 x1862023076615680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.864437:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076615680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.864448:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.864452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.864457:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766829.864465:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766829.864504:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766829.864517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766829/real 1775766829] req@ffffa0434aa7f800 x1862023076615808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766829.864533:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076615808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766829.864540:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766829.864543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766829.864545:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076615808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766830.055087:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766830.055095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766830.887154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766830.887191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766831.079101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766831.079108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766831.911162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766831.911230:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766832.103120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766832.103120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766832.935119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766832.935197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766833.127105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766833.127109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766833.959136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766833.959176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766834.151139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775766834.151139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766834.983146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766834.983180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766834.983182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766834.983188:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983211:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3336 00010000:00010000:1.0:1775766834.983225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766834.983233:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983241:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766834.983281:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076615936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766834.983316:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983321:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983325:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983329:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766834.983350:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076615936:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766834.983356:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076616064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983394:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766834.983396:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983401:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983406:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983410:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983415:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766834.983430:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076616192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983443:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766834.983447:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766834.983448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616064:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766834.983458:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983461:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983465:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983471:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766834.983483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766834.983488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076616320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766834.983504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616320:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766834.983506:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983514:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983525:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983530:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983535:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766834.983559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076616448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766834.983581:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983585:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983588:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983593:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983598:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766834.983612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076616576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766834.983612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616448:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766834.983628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983631:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766834.983633:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983638:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983652:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983657:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983662:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766834.983666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766834.983677:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076616704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983704:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766834.983710:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983715:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00080000:2.0:1775766834.983730:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766834.983732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766834.983738:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983745:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766834.983759:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076616832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766834.983769:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766834.983773:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766834.983775:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766834.983780:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3336 00000100:00100000:3.0:1775766834.983784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616832:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766834.983785:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766834.983794:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766834.983800:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766834.983814:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076616960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766834.983862:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766835.175143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766835.175168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766836.007195:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766836.007258:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766836.199139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766836.199140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766837.031166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766837.031171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766837.223126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766837.223143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766838.055275:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766838.055475:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:1.0:1775766838.055660:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775766838.055719:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766838.055738:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766838.055815:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775766838.055845:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775766838.055891:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7c700 x1862023076616960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766838.055917:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766838.055940:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775766838.055950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076616960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775766838.055962:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766838.055981:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775766838.056006:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775766838.056009:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056084:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056100:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775766838.056128:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775766838.056131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7c380 x1862023076616704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056166:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076616704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056216:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056224:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.056243:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7e680 x1862023076616576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076616576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056283:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056290:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056296:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.056308:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7f480 x1862023076616448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056331:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076616448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056368:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056374:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766838.056386:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7dc00 x1862023076616320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056409:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076616320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056424:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056431:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056436:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766838.056449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7f100 x1862023076616064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076616064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056483:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056495:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.056523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7c000 x1862023076616192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056545:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076616192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056585:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076616192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.056596:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7f800 x1862023076615936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076615936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056648:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076615936:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766838.056701:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766834/real 1775766838] req@ffffa0434aa7e300 x1862023076616832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.056730:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076616832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.056747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.056755:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766838.056762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076616832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766838.247138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766838.247138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766839.079169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766839.079353:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766839.271120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766839.271132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775766840.103139:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103148:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766840.103152:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103178:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103190:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103209:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766840.103246:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076617088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766840.103247:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766840.103276:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103281:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766840.103283:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103289:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103293:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103298:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103308:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766840.103311:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076617088:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766840.103321:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076617216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1775766840.103324:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766840.103337:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103340:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766840.103341:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103345:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103348:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103353:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103359:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766840.103386:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766840.103395:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076617344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766840.103409:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103413:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766840.103415:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103420:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103429:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766840.103434:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7fb80 x1862023076617088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766840.103441:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103453:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766840.103471:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076617472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766840.103477:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076617088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766840.103499:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766840.103504:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766840.103505:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766840.103511:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766840.103516:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766840.103523:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00100000:0.0:1775766840.103528:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076617088:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766840.103532:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103542:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103548:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766840.103565:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076617600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766840.103569:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766840.103592:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103596:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766840.103599:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766840.103616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766840.103621:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103630:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103636:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766840.103640:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7ed80 x1862023076617216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766840.103649:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766840.103667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076617216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766840.103676:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076617728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766840.103687:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.103698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766840.103698:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103708:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766840.103709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766840.103712:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103718:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103723:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103728:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103738:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766840.103746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766840.103755:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076617856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766840.103769:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103773:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766840.103776:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103785:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3341 00000800:00000100:0.0:1775766840.103786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766840.103794:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766840.103801:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766840.103809:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766840.103810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7ce00 x1862023076617344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766840.103829:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076617984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766840.103833:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076617344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.103848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.103853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.103857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766840.103868:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617472:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766840.103893:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766840.103899:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775766840.103906:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766840.103906:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766840.103917:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3341 00000100:00080000:2.0:1775766840.103924:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766840.103927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7df80 x1862023076617472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766840.103944:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766840.103947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076617472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766840.103958:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766840.103963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.103969:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.103975:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766840.103981:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076618112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766840.104008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766840.104073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766840.104091:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7ea00 x1862023076617600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766840.104128:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076617600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.104137:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.104140:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.104144:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076617600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766840.104150:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766840.104186:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766840.104197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7d500 x1862023076617728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766840.104210:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076617728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.104217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.104221:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.104224:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766840.104229:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766840.104249:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766840.104259:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7d880 x1862023076617856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766840.104270:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076617856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.104291:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.104295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.104299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766840.104304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766840.104326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766840.104339:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7fb80 x1862023076617984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766840.104352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076617984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.104360:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.104364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.104367:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076617984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766840.104373:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076618112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766840.104392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766840.104403:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766840/real 1775766840] req@ffffa0434aa7e300 x1862023076618112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766840.104416:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076618112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766840.104425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766840.104428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766840.104440:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076618112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766840.295138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766840.295144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766841.127132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766841.127240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766841.319098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766841.319098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766842.151183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766842.151191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766842.343129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766842.343129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766843.175123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766843.175222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766843.367110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766843.367110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766844.199202:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766844.199269:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766844.391086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766844.391098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766845.223145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766845.223178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766845.223195:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223205:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766845.223209:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223223:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223238:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223247:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766845.223298:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076618240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223332:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223335:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766845.223337:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223340:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223344:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223349:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223359:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766845.223370:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076618368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223378:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223383:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766845.223384:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076618240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766845.223386:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223400:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223424:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223430:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223438:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766845.223454:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076618496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766845.223468:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766845.223479:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223484:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766845.223486:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223492:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223497:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223501:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223507:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766845.223517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349641880 x1862023076618240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766845.223521:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076618624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223573:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223576:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766845.223579:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223583:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223585:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223589:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766845.223590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076618240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766845.223593:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766845.223610:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076618752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766845.223619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.223630:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.223646:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076618240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766845.223656:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223660:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766845.223662:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223668:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223672:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223677:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223681:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766845.223683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766845.223706:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076618880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223717:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223726:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775766845.223727:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766845.223735:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223742:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223746:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223751:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766845.223755:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349641500 x1862023076618368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766845.223758:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766845.223783:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076618368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766845.223788:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076619008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223860:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766845.223861:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766845.223872:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766845.223877:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766845.223877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766845.223892:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00100000:3.0:1775766845.223894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766845.223902:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766845.223907:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223912:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766845.223920:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766845.223926:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076619136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766845.223936:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766845.223939:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766845.223941:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766845.223944:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3346 00000100:00080000:0.0:1775766845.223947:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766845.223949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766845.223955:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766845.223960:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766845.223970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349642a00 x1862023076618496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766845.223974:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076619264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766845.224071:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076618496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224096:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224103:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224123:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766845.224129:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766845.224148:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224156:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349643800 x1862023076618624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224164:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076618624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224174:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766845.224178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766845.224189:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349643b80 x1862023076618752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224209:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076618752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224214:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224217:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224218:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076618752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766845.224234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076618880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766845.224247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224253:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349643100 x1862023076618880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076618880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224266:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224267:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076618880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766845.224281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766845.224293:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349641880 x1862023076619008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224307:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076619008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224328:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224332:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224335:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766845.224342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766845.224365:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349640380 x1862023076619136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224386:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076619136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224392:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224394:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224396:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766845.224400:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766845.224414:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766845.224420:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766845/real 1775766845] req@ffffa04349642300 x1862023076619264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766845.224427:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076619264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766845.224432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766845.224434:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766845.224441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076619264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766845.415086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766845.415091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766846.247187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766846.247265:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766846.439162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766846.439163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766847.271138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766847.271245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766847.463114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766847.463133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766848.295172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766848.295264:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766848.487165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766848.487169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766849.319136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766849.319242:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766849.511155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766849.511157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766850.343124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766850.343141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766850.343151:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343159:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3352 00010000:00010000:3.0:1775766850.343167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766850.343169:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343176:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766850.343209:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076619392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766850.343250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766850.343254:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343258:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343261:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343280:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766850.343282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076619392:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766850.343293:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076619520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766850.343302:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766850.343306:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343309:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343313:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343317:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766850.343324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076619648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766850.343331:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766850.343334:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343336:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343345:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766850.343346:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766850.343352:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343357:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766850.343367:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076619776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766850.343375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343379:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766850.343381:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343386:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00100000:3.0:1775766850.343390:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349641180 x1862023076619392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766850.343392:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343401:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343410:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766850.343415:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076619392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343424:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766850.343428:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076619904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766850.343430:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343436:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076619392:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766850.343449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766850.343453:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343457:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343460:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766850.343463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766850.343467:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343472:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766850.343484:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076620032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766850.343492:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349641c00 x1862023076619520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766850.343515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766850.343521:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766850.343522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076619520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766850.343531:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343537:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343540:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343545:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766850.343555:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766850.343559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076620160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766850.343563:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766850.343579:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766850.343581:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766850.343583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766850.343588:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343592:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343594:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766850.343596:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766850.343609:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766850.343612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076620288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766850.343619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766850.343621:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766850.343622:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766850.343625:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3352 00000100:00080000:0.0:1775766850.343633:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766850.343636:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349640700 x1862023076619648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766850.343640:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766850.343647:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766850.343659:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076620416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766850.343664:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076619648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343679:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343682:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343685:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766850.343691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766850.343709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349642680 x1862023076619776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343726:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076619776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343731:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766850.343746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766850.343759:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349640a80 x1862023076619904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076619904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076619904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766850.343798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766850.343808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349640e00 x1862023076620032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076620032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343827:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343829:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343831:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766850.343834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766850.343843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349641180 x1862023076620160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076620160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766850.343877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766850.343887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349642300 x1862023076620288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076620288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766850.343912:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766850.343921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766850.343927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766850/real 1775766850] req@ffffa04349640380 x1862023076620416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766850.343934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076620416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766850.343944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766850.343946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766850.343948:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076620416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766850.535116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766850.535131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766851.367132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766851.367221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766851.559116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766851.559123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766852.391158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766852.391188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766852.583100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766852.583110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766853.415112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766853.415224:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766853.607150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766853.607241:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766854.439198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766854.439218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766854.631113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766854.631155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766855.463185:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766855.463192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766855.463197:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766855.463203:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463229:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3357 00010000:00010000:0.0:1775766855.463231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766855.463244:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463256:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766855.463301:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076620544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766855.463339:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463342:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766855.463344:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463348:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463352:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463356:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463365:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766855.463378:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076620672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766855.463396:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076620544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766855.463412:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463415:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766855.463417:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463421:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463425:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463429:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463433:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766855.463446:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076620800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766855.463453:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463457:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766855.463458:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463462:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463464:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463467:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463472:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766855.463472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766855.463484:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076620928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766855.463493:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463496:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766855.463497:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463500:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463509:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463513:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463518:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766855.463532:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076621056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766855.463532:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7f800 x1862023076620544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766855.463562:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463567:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766855.463568:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463572:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463576:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463581:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766855.463581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076620544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766855.463588:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766855.463600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766855.463603:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076621184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766855.463608:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.463614:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076620544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766855.463625:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463627:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766855.463629:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463633:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463637:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766855.463639:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766855.463643:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463649:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766855.463660:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076621312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766855.463681:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463684:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766855.463686:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463706:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463711:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766855.463713:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766855.463718:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463723:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766855.463734:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076621440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766855.463742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7c000 x1862023076620672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766855.463747:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766855.463775:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766855.463778:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766855.463783:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3357 00000100:00080000:2.0:1775766855.463786:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766855.463790:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766855.463794:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766855.463803:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076621568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766855.463821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076620672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.463830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.463834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.463838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766855.463844:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766855.463865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.463874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7f100 x1862023076620800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.463884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076620800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.463891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.463893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.463896:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766855.463900:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766855.463921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.463934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7dc00 x1862023076620928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.463949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076620928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.463977:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.463982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.463984:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076620928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766855.463991:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766855.464086:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.464098:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7f480 x1862023076621056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.464118:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076621056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.464126:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.464129:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.464131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766855.464137:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766855.464152:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.464160:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7e680 x1862023076621184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.464175:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076621184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.464182:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.464185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.464194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766855.464211:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766855.464229:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.464237:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7c380 x1862023076621312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.464247:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076621312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.464262:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.464265:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.464268:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766855.464272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766855.464286:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.464293:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7c700 x1862023076621440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.464302:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076621440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.464308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.464311:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.464313:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766855.464317:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766855.464328:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766855.464343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766855/real 1775766855] req@ffffa0434aa7f800 x1862023076621568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766855.464352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076621568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766855.464358:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766855.464361:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766855.464363:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076621568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766855.655150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766855.655151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766856.487108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766856.487164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766856.679169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766856.679191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766857.511123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766857.511182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766857.703124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766857.703144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766858.535105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766858.535201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766858.727091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766858.727146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766859.559117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766859.559183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766859.751113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766859.751118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766860.583135:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583144:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766860.583148:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583160:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583171:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766860.583177:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775766860.583200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766860.583205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766860.583220:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076621696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766860.583255:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583262:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766860.583266:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583282:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583288:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766860.583292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076621696:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766860.583300:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583311:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766860.583332:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076621824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766860.583344:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583346:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766860.583348:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583352:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583361:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766860.583375:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775766860.583376:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583396:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766860.583424:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076621952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766860.583436:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7e300 x1862023076621696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583437:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583455:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766860.583462:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583476:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583483:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766860.583485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076621696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583494:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583512:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766860.583516:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.583522:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766860.583528:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076622080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766860.583533:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076621696:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766860.583556:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583559:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766860.583561:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583566:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583568:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766860.583570:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583578:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766860.583580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766860.583592:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076622208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766860.583602:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583609:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1775766860.583610:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766860.583616:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583622:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00100000:0.0:1775766860.583625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7fb80 x1862023076621824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583633:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766860.583648:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583654:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766860.583656:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076621824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766860.583666:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076622336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766860.583747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766860.583748:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583757:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766860.583761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766860.583763:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766860.583773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621824:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766860.583778:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583791:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766860.583796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766860.583797:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.583812:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766860.583824:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076622464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766860.583835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766860.583842:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583848:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766860.583851:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7d880 x1862023076621952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583852:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766860.583869:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00100000:0.0:1775766860.583874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076621952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583877:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766860.583892:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766860.583893:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775766860.583901:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.583905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076621952:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766860.583905:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766860.583910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766860.583933:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076622592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766860.583938:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.583960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7d500 x1862023076622080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583971:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766860.583978:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766860.583980:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766860.583987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076622080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766860.583988:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3362 00000100:00080000:2.0:1775766860.583996:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766860.584004:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766860.584013:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775766860.584015:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584069:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766860.584070:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076622720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766860.584103:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766860.584122:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.584131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7ea00 x1862023076622208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766860.584139:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076622208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766860.584146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584148:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584150:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766860.584154:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766860.584165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.584171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7df80 x1862023076622336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766860.584178:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076622336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766860.584184:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584186:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766860.584191:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766860.584203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.584221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7ce00 x1862023076622464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766860.584234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076622464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766860.584244:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584247:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584251:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766860.584258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766860.584276:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.584284:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7ed80 x1862023076622592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766860.584292:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076622592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766860.584297:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584300:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584301:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766860.584305:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766860.584315:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766860.584321:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766860/real 1775766860] req@ffffa0434aa7e300 x1862023076622720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766860.584328:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076622720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766860.584338:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766860.584341:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766860.584343:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076622720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766860.775089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766860.775098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766861.607169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766861.607191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766861.799124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766861.799134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766862.631114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766862.631131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766862.823086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766862.823091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766863.655110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766863.655159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766863.847147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766863.847150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766864.679114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766864.679211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766864.871096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766864.871143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766865.703176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766865.703184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766865.703186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766865.703194:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703204:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703215:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703221:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766865.703234:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766865.703259:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076622848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766865.703307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703311:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766865.703313:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703317:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703323:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766865.703338:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076622848:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766865.703338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766865.703369:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076622976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766865.703381:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766865.703386:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703390:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703395:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703398:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703404:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766865.703405:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766865.703418:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076623104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766865.703428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766865.703433:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703437:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00100000:0.0:1775766865.703446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7f800 x1862023076622848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766865.703452:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703458:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703467:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766865.703476:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076622848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766865.703483:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076623232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766865.703488:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.703498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.703504:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076622848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766865.703506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766865.703513:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703518:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703523:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766865.703528:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766865.703528:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703537:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766865.703550:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076623360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766865.703558:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766865.703560:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766865.703565:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703570:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00100000:0.0:1775766865.703572:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7c700 x1862023076622976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766865.703574:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703581:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703588:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766865.703592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076622976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766865.703599:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076623488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766865.703628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.703635:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.703643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076622976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766865.703648:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703651:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766865.703652:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766865.703654:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703662:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703666:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703669:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703674:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766865.703681:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766865.703685:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076623616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766865.703693:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766865.703697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7c380 x1862023076623104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766865.703697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766865.703698:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703705:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:2.0:1775766865.703710:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766865.703715:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766865.703720:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766865.703728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076623104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766865.703729:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076623744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766865.703759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766865.703762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766865.703763:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766865.703765:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3367 00000100:00080000:0.0:1775766865.703769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766865.703773:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766865.703776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766865.703780:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766865.703783:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623104:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766865.703786:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766865.703791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766865.703806:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076623872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766865.703817:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.703834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7e680 x1862023076623232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.703847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076623232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.703857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.703860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.703871:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766865.703887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766865.703920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.703928:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7f480 x1862023076623360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.703936:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076623360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.703942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.703945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.703947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076623360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766865.703951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766865.703962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.703967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7dc00 x1862023076623488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.703975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076623488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.703980:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.703981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.703983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766865.703986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766865.703995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.704007:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7f100 x1862023076623616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.704015:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076623616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.704020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.704022:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.704024:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766865.704052:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766865.704073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.704085:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7c000 x1862023076623744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.704098:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076623744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.704106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.704109:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.704111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766865.704116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766865.704130:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766865.704136:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766865/real 1775766865] req@ffffa0434aa7f800 x1862023076623872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766865.704144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076623872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766865.704155:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766865.704157:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766865.704159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076623872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766865.895131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766865.895131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766866.727105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766866.727154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766866.919178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766866.919178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766867.751161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766867.751200:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766867.943110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766867.943123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766868.775157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766868.775197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766868.967150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766868.967151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766869.799131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766869.799189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766869.991134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766869.991137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766870.823159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766870.823160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766870.823174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766870.823180:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775766870.823180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766870.823195:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00080000:0.0:1775766870.823208:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823216:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766870.823267:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076624000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766870.823301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766870.823307:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823312:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00080000:0.0:1775766870.823316:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823322:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823334:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766870.823340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076624000:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766870.823352:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076624128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766870.823384:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766870.823389:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823394:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00080000:0.0:1775766870.823408:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766870.823415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766870.823416:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823424:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766870.823438:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076624256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766870.823448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766870.823455:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823459:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00100000:3.0:1775766870.823463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349641880 x1862023076624000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766870.823464:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823473:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823482:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766870.823488:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076624000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766870.823497:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076624384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766870.823497:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.823504:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.823512:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076624000:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766870.823520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766870.823526:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823532:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00100000:3.0:1775766870.823541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766870.823545:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823552:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823558:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775766870.823562:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.823573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349643100 x1862023076624128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766870.823579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076624512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766870.823588:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076624128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766870.823598:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766870.823600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766870.823607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766870.823607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766870.823612:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766870.823613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766870.823624:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00100000:3.0:1775766870.823624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766870.823634:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823640:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766870.823646:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823650:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766870.823654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349643b80 x1862023076624256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766870.823668:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076624640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766870.823676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076624256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766870.823687:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766870.823687:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.823694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766870.823697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766870.823699:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766870.823701:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766870.823708:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624384:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766870.823709:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00080000:0.0:1775766870.823717:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823722:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766870.823728:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766870.823729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.823739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349643800 x1862023076624384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766870.823746:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076624768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766870.823752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076624384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766870.823768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.823771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.823774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766870.823792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624512:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766870.823793:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823799:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766870.823801:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823807:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3372 00000800:00000100:3.0:1775766870.823809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766870.823817:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766870.823823:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766870.823825:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349642a00 x1862023076624512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766870.823832:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766870.823865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076624896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766870.823867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076624512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766870.823890:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766870.823894:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766870.823897:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766870.823906:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3372 00000100:00080000:0.0:1775766870.823915:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766870.823915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766870.823924:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766870.823924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.823930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076624512:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775766870.823934:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766870.823937:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766870.823951:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076625024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766870.823961:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.823973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349641500 x1862023076624640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766870.823982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076624640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766870.823988:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.823991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.823993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766870.824003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766870.824013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.824019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349641880 x1862023076624768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766870.824045:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076624768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766870.824051:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.824053:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.824061:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766870.824067:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766870.824084:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.824091:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349640380 x1862023076624896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766870.824099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076624896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766870.824106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.824109:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.824112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076624896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766870.824116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766870.824126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766870.824131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766870/real 1775766870] req@ffffa04349642300 x1862023076625024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766870.824139:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076625024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766870.824144:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766870.824146:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766870.824147:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766871.015137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766871.015138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766871.847091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766871.847161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766872.039102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766872.039106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766872.871092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766872.871139:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766873.063108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766873.063109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766873.895139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766873.895190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766874.087107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766874.087129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766874.919135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766874.919222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766875.111089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766875.111115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766875.943151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766875.943174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766875.943184:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943191:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943216:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943223:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766875.943223:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766875.943264:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076625152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766875.943294:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943297:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943299:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943302:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943310:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766875.943321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076625280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766875.943336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076625152:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766875.943337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766875.943340:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943355:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943359:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943364:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766875.943387:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076625408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766875.943411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943414:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943419:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943428:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766875.943429:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943442:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766875.943456:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076625536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943466:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766875.943468:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943472:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943477:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943482:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943488:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766875.943492:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349641180 x1862023076625152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766875.943502:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076625664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766875.943524:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766875.943526:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076625152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766875.943532:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943544:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943554:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766875.943556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.943563:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.943572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076625152:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766875.943573:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766875.943597:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076625792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766875.943609:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943618:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00100000:3.0:1775766875.943619:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766875.943622:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766875.943631:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943634:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766875.943650:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076625920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766875.943662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766875.943664:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943666:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00080000:0.0:1775766875.943669:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766875.943675:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766875.943679:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943685:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766875.943695:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076626048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766875.943704:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766875.943706:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349640e00 x1862023076625280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766875.943707:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766875.943718:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766875.943732:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3377 00000100:00100000:3.0:1775766875.943735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076625280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.943745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.943750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766875.943751:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766875.943757:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766875.943763:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766875.943770:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766875.943800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766875.943811:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076626176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766875.943826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.943837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349640a80 x1862023076625408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.943847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076625408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.943857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.943860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.943864:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766875.943872:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766875.943895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.943905:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349642680 x1862023076625536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.943913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076625536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.943919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.943922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.943924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766875.943928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766875.943941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.943947:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349640700 x1862023076625664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.943961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076625664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.943967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.943970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.943971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076625664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766875.943976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766875.943989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.943996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349641c00 x1862023076625792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.944003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076625792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.944009:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.944011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.944013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766875.944050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766875.944063:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.944070:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349641180 x1862023076625920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.944078:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076625920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.944100:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.944103:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.944105:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076625920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766875.944110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766875.944124:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.944131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349642300 x1862023076626048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.944139:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076626048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.944145:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.944147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.944149:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766875.944153:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766875.944162:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766875.944167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766875/real 1775766875] req@ffffa04349640380 x1862023076626176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766875.944174:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076626176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766875.944179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766875.944181:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766875.944188:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766876.135110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766876.135138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766876.967171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766876.967245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766877.159156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766877.159156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766877.991141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766877.991220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766878.183124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766878.183124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766879.015157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766879.015173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766879.207171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766879.207190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766880.039161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766880.039205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766880.231137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766880.231137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766881.063163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766881.063164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766881.063175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766881.063181:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063196:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3382 00010000:00010000:3.0:1775766881.063205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766881.063210:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063222:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766881.063275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076626304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766881.063306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766881.063312:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063318:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:0.0:1775766881.063325:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063331:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063344:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766881.063353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076626304:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766881.063362:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076626432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766881.063405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766881.063409:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063416:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:0.0:1775766881.063422:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766881.063427:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766881.063432:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766881.063465:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076626560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766881.063471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349641880 x1862023076626304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766881.063503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766881.063510:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063513:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:0.0:1775766881.063518:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766881.063524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076626304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766881.063530:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766881.063550:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.063556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766881.063562:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076626688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766881.063565:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076626304:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766881.063587:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766881.063592:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063596:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00100000:3.0:1775766881.063609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766881.063617:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063623:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063628:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766881.063640:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076626816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766881.063649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766881.063662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766881.063665:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349641500 x1862023076626432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766881.063670:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766881.063677:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063692:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:0.0:1775766881.063700:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063704:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766881.063709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076626432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766881.063715:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766881.063751:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076626944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766881.063777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063791:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766881.063794:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063804:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:3.0:1775766881.063809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766881.063810:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063819:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766881.063826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766881.063829:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766881.063836:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766881.063842:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076627072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766881.063851:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766881.063872:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063876:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766881.063879:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063887:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3382 00000800:00000100:3.0:1775766881.063892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766881.063902:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063909:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063915:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766881.063916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349642a00 x1862023076626560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766881.063931:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076627200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766881.063938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076626560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766881.063948:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766881.063961:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766881.063966:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766881.063979:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3382 00000100:00080000:3.0:1775766881.063982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766881.063984:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766881.063990:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766881.063995:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766881.063995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766881.064009:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076627328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766881.064060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766881.064108:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064123:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349643800 x1862023076626688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064133:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076626688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064147:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064152:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064161:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766881.064192:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766881.064217:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064230:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349643b80 x1862023076626816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064242:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076626816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064248:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064250:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064252:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076626816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766881.064256:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766881.064271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064279:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349643100 x1862023076626944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064286:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076626944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064292:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064294:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064296:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076626944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766881.064299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766881.064309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064324:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349641880 x1862023076627072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064331:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076627072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064337:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064339:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064341:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766881.064345:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766881.064356:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349640380 x1862023076627200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064368:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076627200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064373:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064375:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064377:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766881.064380:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766881.064390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766881.064395:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766881/real 1775766881] req@ffffa04349642300 x1862023076627328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766881.064402:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076627328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766881.064411:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766881.064413:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766881.064415:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076627328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766881.255129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766881.255182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766882.087122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766882.087439:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766882.279112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766882.279152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766883.111086:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766883.111138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766883.303125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766883.303129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766884.135126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766884.135176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766884.327101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766884.327102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766885.159099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766885.159112:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766885.351115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766885.351115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766886.183098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766886.183139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766886.183215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766886.183223:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183231:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183240:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183245:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766886.183275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076627456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766886.183334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076627456:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775766886.183391:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766886.183422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766886.183429:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183434:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183439:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183443:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183453:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766886.183468:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076627584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766886.183528:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349641180 x1862023076627456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.183541:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076627456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.183549:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.183551:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.183554:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076627456:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766886.183573:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766886.183590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.183599:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349641c00 x1862023076627584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.183623:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076627584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.183630:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.183633:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.183636:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766886.183703:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766886.183713:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183716:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183721:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183724:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183729:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766886.183746:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076627712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766886.183759:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183762:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766886.183764:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183766:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183769:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183772:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183776:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766886.183783:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076627840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766886.183791:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766886.183792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766886.183794:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766886.183797:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183805:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183806:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183809:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183812:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766886.183821:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076627968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766886.183827:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766886.183831:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183838:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183842:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766886.183843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766886.183850:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183856:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766886.183864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349640700 x1862023076627712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766886.183870:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076628096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766886.183893:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766886.183897:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076627712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766886.183898:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766886.183902:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.183910:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183917:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766886.183918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766886.183922:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766886.183926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775766886.183927:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766886.183934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766886.183940:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076628224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766886.183947:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183950:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766886.183952:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766886.183954:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627840:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766886.183957:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00080000:0.0:1775766886.183962:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.183965:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.183968:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766886.183977:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076628352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766886.183988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766886.183995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766886.183998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766886.183999:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766886.184004:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3387 00000100:00100000:3.0:1775766886.184004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349642680 x1862023076627840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766886.184010:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766886.184016:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766886.184023:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766886.184056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076627840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766886.184061:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076628480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766886.184092:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184096:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184099:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766886.184105:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766886.184138:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.184152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349640a80 x1862023076627968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.184165:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076627968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.184173:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184176:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184178:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076627968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766886.184182:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766886.184197:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.184204:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349640e00 x1862023076628096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.184212:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076628096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.184217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184219:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184226:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766886.184230:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766886.184243:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.184249:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349641c00 x1862023076628224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.184256:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076628224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.184262:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184264:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184266:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766886.184278:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766886.184287:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.184293:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349641180 x1862023076628352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.184300:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076628352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.184305:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184306:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184308:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766886.184312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766886.184320:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766886.184331:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766886/real 1775766886] req@ffffa04349642300 x1862023076628480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766886.184339:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076628480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766886.184344:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766886.184345:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766886.184347:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076628480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766886.375087:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766886.375105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766887.207107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766887.207122:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766887.399107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766887.399120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766888.231097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766888.231156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766888.423142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766888.423142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766889.255127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766889.255130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775766889.447123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766889.447133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766890.279175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766890.279264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766890.471134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766890.471134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766891.303119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766891.303142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775766891.303151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766891.303152:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303161:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303170:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303175:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766891.303206:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076628608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766891.303228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766891.303248:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303252:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303256:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766891.303257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076628608:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766891.303262:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766891.303271:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766891.303284:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076628736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766891.303292:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303295:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766891.303296:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303299:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303302:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303304:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766891.303308:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766891.303318:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766891.303319:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076628864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766891.303328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766891.303333:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303336:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303347:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303350:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303355:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349640380 x1862023076628608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766891.303362:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766891.303374:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076628992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766891.303375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076628608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766891.303403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766891.303406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303410:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:3.0:1775766891.303411:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766891.303414:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766891.303417:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766891.303420:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766891.303424:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766891.303425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076628608:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766891.303434:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076629120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766891.303448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766891.303452:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303455:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303457:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303459:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766891.303462:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766891.303466:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766891.303470:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076629248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766891.303478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766891.303488:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766891.303490:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766891.303494:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303499:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303504:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349641880 x1862023076628736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766891.303510:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766891.303521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076628736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766891.303526:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076629376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766891.303532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303540:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766891.303549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766891.303552:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766891.303557:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766891.303560:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303566:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303572:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766891.303572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766891.303579:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349643100 x1862023076628864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766891.303586:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766891.303594:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076628864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303602:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303604:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766891.303611:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076629504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766891.303616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628992:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766891.303640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766891.303643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766891.303644:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766891.303647:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3392 00000100:00080000:0.0:1775766891.303651:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766891.303653:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766891.303658:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766891.303666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766891.303678:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076629632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766891.303684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349643b80 x1862023076628992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076628992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076628992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766891.303732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076629120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766891.303749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349643800 x1862023076629120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076629120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303781:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076629120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766891.303799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766891.303810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303816:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349642a00 x1862023076629248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076629248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766891.303857:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766891.303869:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303876:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349641500 x1862023076629376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076629376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766891.303895:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766891.303904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349640380 x1862023076629504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076629504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303929:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766891.303933:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766891.303945:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766891.303952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766891/real 1775766891] req@ffffa04349642300 x1862023076629632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766891.303964:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076629632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766891.303972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766891.303975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766891.303978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076629632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766891.495087:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766891.495090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766892.327114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766892.327173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766892.519124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766892.519126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766893.351165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766893.351187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766893.543159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766893.543159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766894.375148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766894.375230:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766894.567163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766894.567163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766895.399152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766895.399227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766895.591139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766895.591139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766896.423201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766896.423223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775766896.423242:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766896.423257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766896.423265:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423291:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423311:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423323:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766896.423410:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076629760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766896.423449:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423485:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423495:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423505:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423523:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766896.423527:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076629760:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766896.423602:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076629888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423661:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766896.423663:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423668:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423680:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423687:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423694:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766896.423713:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076630016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423730:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766896.423732:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423735:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423738:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423753:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423757:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766896.423765:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076630144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423772:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766896.423777:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423781:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423783:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423786:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423796:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766896.423805:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076630272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423813:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766896.423818:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423821:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423824:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423827:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423830:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766896.423837:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076630400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423844:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423847:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766896.423849:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423852:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423854:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423857:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423860:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766896.423867:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076630528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423875:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766896.423887:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423892:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423895:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423899:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423903:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766896.423912:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076630656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766896.423920:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766896.423923:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766896.423925:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766896.423928:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3398 00000100:00080000:0.0:1775766896.423931:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766896.423934:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766896.423937:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766896.423945:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076630784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766896.424211:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424286:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349641180 x1862023076629760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076629760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424350:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424356:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076629760:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775766896.424403:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076629888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766896.424435:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424450:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349641c00 x1862023076629888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424493:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076629888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424502:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424506:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424511:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076629888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766896.424521:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766896.424562:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424600:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349640e00 x1862023076630016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076630016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424664:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766896.424690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766896.424732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349640a80 x1862023076630144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076630144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766896.424833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766896.424861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349642680 x1862023076630272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076630272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424893:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076630272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766896.424906:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766896.424922:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349640700 x1862023076630400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.424952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076630400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.424960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.424964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.424967:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766896.424972:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766896.424990:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.424999:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349642300 x1862023076630528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.425010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076630528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.425017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.425020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.425022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766896.425089:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766896.425148:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.425171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349640380 x1862023076630656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.425216:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076630656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.425238:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.425242:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.425245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766896.425252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766896.425300:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766896.425313:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766896/real 1775766896] req@ffffa04349641500 x1862023076630784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766896.425328:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076630784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766896.425338:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766896.425342:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766896.425346:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076630784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766896.615149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766896.615149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766897.447157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766897.447171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766897.639094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766897.639094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766898.471151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766898.471165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766898.663102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766898.663102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766899.495119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766899.495138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766899.687187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766899.687187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766900.519188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766900.519222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766900.711152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766900.711164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766901.543108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766901.543113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766901.543116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766901.543122:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543130:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3403 00010000:00010000:3.0:1775766901.543132:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766901.543154:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543167:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766901.543197:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076630912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766901.543223:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543226:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543229:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543237:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543245:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766901.543251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076630912:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766901.543256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076631040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766901.543272:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543275:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543278:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543280:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543283:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766901.543292:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076631168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543307:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766901.543309:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543312:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543316:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543319:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543323:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766901.543331:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076631296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766901.543343:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543346:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543349:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543351:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543355:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766901.543363:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766901.543364:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076631424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766901.543381:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543385:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543387:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543390:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766901.543399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631168:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766901.543401:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766901.543416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076631552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766901.543421:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631296:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766901.543430:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766901.543435:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766901.543438:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543443:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766901.543447:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631424:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766901.543448:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766901.543483:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076631680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543495:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766901.543498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766901.543499:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631552:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766901.543512:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543516:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543539:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543563:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766901.543578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766901.543579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076631808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766901.543596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766901.543601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766901.543610:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766901.543610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631808:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766901.543621:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3403 00000100:00080000:0.0:1775766901.543625:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766901.543629:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766901.543634:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766901.543646:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076631936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766901.543687:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766901.735106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775766901.735115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766902.567174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766902.567263:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775766902.759127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766902.759167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766903.591225:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766903.591306:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766903.783146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766903.783167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766904.615202:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766904.615346:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:3.0:1775766904.615381:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775766904.615438:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766904.615463:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766904.615561:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775766904.615610:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766904.615611:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349640700 x1862023076631936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775766904.615651:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766904.615663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076631936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.615679:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:3.0:1775766904.615680:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:0.0:1775766904.615692:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.615698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766904.615713:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766904.615714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349643800 x1862023076631040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.615756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076631040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.615764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:3.0:1775766904.615773:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:0.0:1775766904.615777:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.615812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766904.615827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349643b80 x1862023076631168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775766904.615842:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766904.615873:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076631168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775766904.615887:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:0.0:1775766904.615915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.615926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.615934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766904.615944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349643100 x1862023076631296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.615963:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076631296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.615972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.615976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.615980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766904.615990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349642a00 x1862023076630912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.616005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076630912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.616014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.616018:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.616081:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076630912:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766904.616141:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349642300 x1862023076631808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.616158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076631808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.616168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.616172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.616176:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766904.616185:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349640380 x1862023076631680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.616200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076631680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.616209:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.616213:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.616217:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766904.616227:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349641880 x1862023076631424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.616247:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076631424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.616258:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.616262:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.616275:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076631424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766904.616286:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766901/real 1775766904] req@ffffa04349641500 x1862023076631552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766904.616303:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076631552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766904.616313:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766904.616317:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766904.616321:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076631552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766904.807098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766904.807125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766905.639173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766905.639274:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766905.831126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766905.831135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766906.663123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766906.663132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766906.663136:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663146:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3408 00010000:00010000:3.0:1775766906.663149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766906.663156:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766906.663164:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766906.663200:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076632064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766906.663227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766906.663228:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766906.663232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766906.663240:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663253:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663257:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766906.663267:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076632064:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766906.663278:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663288:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766906.663301:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076632192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766906.663312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766906.663315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766906.663317:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663321:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663325:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766906.663330:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766906.663339:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766906.663356:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076632320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766906.663364:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766906.663367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766906.663369:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766906.663372:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349641500 x1862023076632064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766906.663375:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663381:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766906.663388:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663394:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766906.663400:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076632064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766906.663408:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076632448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766906.663412:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.663420:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766906.663427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766906.663427:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076632064:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766906.663441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766906.663444:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766906.663448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632192:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766906.663451:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663458:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766906.663476:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663483:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775766906.663497:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766906.663502:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076632576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766906.663516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766906.663516:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349641880 x1862023076632192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766906.663527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766906.663532:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663542:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663548:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766906.663553:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076632192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766906.663557:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663566:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766906.663570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.663576:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.663581:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766906.663588:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076632704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766906.663592:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766906.663600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766906.663611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766906.663612:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663618:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663623:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766906.663628:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766906.663636:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663647:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766906.663654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349640380 x1862023076632320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766906.663664:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076632832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766906.663677:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766906.663680:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766906.663682:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663687:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663693:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766906.663696:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076632320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766906.663700:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663709:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766906.663720:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766906.663726:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076632960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766906.663729:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.663738:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766906.663750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766906.663753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632448:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766906.663758:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766906.663763:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766906.663768:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3408 00000100:00080000:0.0:1775766906.663773:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766906.663778:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766906.663786:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766906.663790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766906.663805:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076633088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766906.663808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349642300 x1862023076632448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.663823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076632448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.663840:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.663847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.663852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766906.663878:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766906.663901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766906.663914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349642a00 x1862023076632576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.663943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076632576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.663953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.663957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.663960:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076632576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766906.663966:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766906.664105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766906.664123:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349643100 x1862023076632704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.664138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076632704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.664147:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.664151:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.664154:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766906.664160:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766906.664183:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766906.664195:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349643b80 x1862023076632832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.664206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076632832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.664228:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.664232:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.664236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766906.664241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766906.664264:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766906.664277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349643800 x1862023076632960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.664289:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076632960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.664297:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.664301:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.664304:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076632960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766906.664311:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766906.664337:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766906.664348:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766906/real 1775766906] req@ffffa04349640700 x1862023076633088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766906.664362:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076633088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766906.664369:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766906.664373:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766906.664383:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766906.855119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766906.855127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775766907.687282:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766907.687407:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766907.879107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766907.879107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766908.711126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766908.711143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766908.903100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766908.903100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766909.735080:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766909.735151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766909.927111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766909.927118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766910.759143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766910.759286:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766910.951152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766910.951185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766911.783116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766911.783127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1775766911.783132:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766911.783133:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766911.783137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766911.783145:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783156:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783162:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766911.783195:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076633216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766911.783220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766911.783221:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783224:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783227:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783229:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783237:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766911.783246:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076633344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766911.783255:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076633216:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766911.783256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766911.783263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766911.783265:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783268:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783285:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783289:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783293:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766911.783302:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076633472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783309:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775766911.783314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766911.783319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766911.783321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783326:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783332:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783335:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766911.783349:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349642680 x1862023076633216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766911.783351:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076633600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766911.783378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766911.783379:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766911.783382:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076633216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766911.783386:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783391:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783396:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766911.783399:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766911.783400:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775766911.783406:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766911.783423:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076633728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766911.783431:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076633216:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766911.783444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766911.783447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766911.783448:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783453:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783457:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766911.783457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766911.783464:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766911.783481:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766911.783483:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076633856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766911.783494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349640a80 x1862023076633344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766911.783496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766911.783497:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783500:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00080000:0.0:1775766911.783502:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783506:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783512:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766911.783524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076633344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766911.783525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076633984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783557:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766911.783558:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766911.783564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766911.783565:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766911.783566:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766911.783570:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766911.783573:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00100000:3.0:1775766911.783579:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766911.783582:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783586:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783593:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766911.783604:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076634112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766911.783611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775766911.783612:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766911.783617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766911.783619:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766911.783623:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3413 00000100:00100000:3.0:1775766911.783626:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349640e00 x1862023076633472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766911.783627:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766911.783634:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766911.783641:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766911.783645:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076633472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766911.783651:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076634240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766911.783653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783661:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783667:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766911.783673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766911.783686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783692:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349641c00 x1862023076633600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076633600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783707:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766911.783710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766911.783719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349641180 x1862023076633728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076633728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783752:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076633728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766911.783765:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766911.783776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349642680 x1862023076633856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783798:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076633856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783808:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766911.783811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766911.783821:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349640700 x1862023076633984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783833:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076633984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076633984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766911.783855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076634112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766911.783867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349643800 x1862023076634112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076634112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783892:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076634112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766911.783897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076634240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766911.783908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766911.783913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766911/real 1775766911] req@ffffa04349643b80 x1862023076634240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766911.783920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076634240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766911.783924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766911.783926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766911.783932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076634240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766911.975087:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766911.975100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766912.807119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766912.807131:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766912.999126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766912.999126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766913.831209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766913.831287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766914.023147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766914.023149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766914.855132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766914.855190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766915.047197:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766915.047215:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766915.879121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766915.879219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766916.071180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766916.071224:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766916.903121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766916.903139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775766916.903142:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766916.903147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766916.903152:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903162:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903171:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903196:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766916.903224:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076634368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903253:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766916.903259:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903264:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903274:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766916.903292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076634368:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766916.903301:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076634496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903313:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903317:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766916.903320:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903324:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903334:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903340:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766916.903346:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766916.903355:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076634624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903381:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766916.903387:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903390:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903395:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903398:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766916.903404:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349643100 x1862023076634368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766916.903416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076634752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903425:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903427:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766916.903428:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766916.903432:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076634368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766916.903434:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903439:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903445:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903452:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775766916.903453:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903458:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766916.903461:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076634880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766916.903465:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076634368:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766916.903476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766916.903479:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903483:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903485:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903488:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903491:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766916.903497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766916.903507:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076635008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903523:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766916.903525:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766916.903530:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766916.903534:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903541:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903550:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349642a00 x1862023076634496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766916.903554:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766916.903567:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076634496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766916.903573:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076635136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766916.903576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903585:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766916.903591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766916.903591:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634496:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766916.903597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766916.903600:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766916.903603:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00080000:0.0:1775766916.903616:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903622:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903626:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766916.903626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766916.903638:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076635264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766916.903645:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766916.903647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766916.903649:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766916.903652:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766916.903661:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3418 00000100:00100000:3.0:1775766916.903663:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349642300 x1862023076634624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766916.903666:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766916.903673:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766916.903679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766916.903680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076634624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.903689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766916.903691:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076635392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766916.903695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.903702:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766916.903711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766916.903726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903732:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349640380 x1862023076634752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.903739:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076634752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.903744:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903745:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.903747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766916.903751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766916.903760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903768:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349641880 x1862023076634880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.903802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076634880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.903810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903813:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.903816:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076634880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766916.903835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766916.903856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349641500 x1862023076635008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.903878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076635008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.903886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.903892:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766916.903896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766916.903914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349643100 x1862023076635136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.903934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076635136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.903953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.903957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.903961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766916.903967:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766916.903988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.903996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349643b80 x1862023076635264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.904013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076635264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.904018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.904020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.904022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766916.904027:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766916.904065:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766916.904076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766916/real 1775766916] req@ffffa04349643800 x1862023076635392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766916.904089:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076635392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766916.904097:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766916.904100:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766916.904109:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076635392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766917.095114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766917.095114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766917.927120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766917.927169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766918.119183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766918.119247:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766918.951101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766918.951138:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766919.143137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766919.143138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766919.975156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766919.975255:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766920.167127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766920.167127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766920.999196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766920.999196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766921.191118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766921.191128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766922.023169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766922.023207:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023217:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1775766922.023219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766922.023223:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023236:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023250:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023258:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766922.023327:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076635520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023356:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023359:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766922.023361:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023365:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023369:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023374:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766922.023401:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076635520:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775766922.023411:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766922.023429:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076635648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023441:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023444:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766922.023446:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023450:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023454:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023458:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023463:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766922.023473:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766922.023477:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076635776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023488:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023491:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766922.023493:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023498:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023500:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023503:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023508:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766922.023518:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349640700 x1862023076635520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766922.023519:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076635904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023605:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023608:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766922.023612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076635520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766922.023621:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766922.023628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766922.023648:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:3.0:1775766922.023648:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766922.023666:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766922.023667:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076635520:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766922.023679:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023686:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766922.023700:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076636032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766922.023703:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766922.023730:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023733:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766922.023735:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023741:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023744:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766922.023749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766922.023750:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023758:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766922.023769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349642680 x1862023076635648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766922.023772:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076636160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023812:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023823:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766922.023826:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023833:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023837:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023843:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766922.023845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076635648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766922.023849:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766922.023862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766922.023874:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076636288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766922.023874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.023894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766922.023896:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023910:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766922.023915:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023920:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023924:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766922.023925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635776:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766922.023929:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023934:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766922.023945:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076636416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766922.023959:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766922.023962:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766922.023964:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766922.023970:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3423 00000100:00080000:0.0:1775766922.023974:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766922.023978:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766922.023985:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766922.023988:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766922.024010:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076636544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766922.024011:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349641180 x1862023076635776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024086:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076635776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766922.024114:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766922.024138:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349641c00 x1862023076635904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076635904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024190:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024195:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024197:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076635904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766922.024211:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766922.024222:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024228:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349640e00 x1862023076636032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024245:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076636032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024251:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024253:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024255:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766922.024259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766922.024271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349640a80 x1862023076636160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024285:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076636160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024289:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024291:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024293:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766922.024297:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766922.024321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024328:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349640700 x1862023076636288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024335:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076636288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024350:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024353:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024355:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766922.024359:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766922.024373:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024380:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349643800 x1862023076636416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076636416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024392:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024393:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024395:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766922.024399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766922.024408:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766922.024413:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766922/real 1775766922] req@ffffa04349643b80 x1862023076636544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766922.024420:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076636544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766922.024424:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766922.024426:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766922.024432:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076636544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766922.215154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766922.215154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766923.047122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766923.047150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766923.239101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766923.239105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766924.071113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766924.071157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766924.263141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766924.263141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766925.095173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766925.095174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766925.287122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766925.287122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766926.119140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766926.119231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766926.311125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766926.311126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766927.143184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766927.143201:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775766927.143201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766927.143216:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143230:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143239:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766927.143270:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766927.143342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076636672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143370:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766927.143375:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143379:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143383:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143387:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766927.143407:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076636672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766927.143412:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076636800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143421:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766927.143425:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143428:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143431:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143465:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143470:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766927.143479:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076636928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143486:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143490:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766927.143491:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143493:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143513:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766927.143517:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766927.143522:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143527:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766927.143537:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076637056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143544:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766927.143548:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143552:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143555:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143558:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766927.143569:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349643100 x1862023076636672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766927.143574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076637184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766927.143606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076636672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766927.143608:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766927.143613:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143624:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:3.0:1775766927.143627:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766927.143629:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766927.143636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766927.143637:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143645:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766927.143646:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076636672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766927.143662:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076637312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143671:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143674:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766927.143678:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766927.143678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766927.143695:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143699:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143702:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143706:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766927.143718:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076637440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766927.143722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766927.143734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766927.143737:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766927.143740:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766927.143740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349641500 x1862023076636800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766927.143749:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143756:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143762:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766927.143767:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076636800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766927.143793:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766927.143806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.143811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766927.143816:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076637568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766927.143824:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766927.143827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766927.143829:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766927.143833:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766927.143837:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3428 00000100:00080000:0.0:1775766927.143840:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766927.143848:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766927.143851:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766927.143854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766927.143862:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076637696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766927.143886:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.143898:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349641880 x1862023076636928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.143909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076636928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.143916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.143920:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.143924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076636928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766927.143932:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766927.143947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.143956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349640380 x1862023076637056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.143965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076637056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.143979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.143981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.143982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766927.143987:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766927.143998:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.144004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349642300 x1862023076637184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.144021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076637184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.144060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.144082:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.144085:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766927.144090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766927.144111:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.144123:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349642a00 x1862023076637312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.144133:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076637312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.144139:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.144142:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.144144:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766927.144162:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766927.144174:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.144181:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349643100 x1862023076637440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.144189:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076637440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.144206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.144209:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.144212:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766927.144216:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766927.144231:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.144239:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349643b80 x1862023076637568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.144248:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076637568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.144271:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.144274:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.144277:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766927.144282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766927.144312:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766927.144327:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766927/real 1775766927] req@ffffa04349643800 x1862023076637696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766927.144336:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076637696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766927.144341:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766927.144343:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766927.144351:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076637696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766927.335163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766927.335164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766928.167140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766928.167200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766928.359109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766928.359113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766929.191117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766929.191142:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766929.383126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766929.383129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766930.215127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766930.215151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766930.407108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766930.407108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766931.239110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766931.239159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766931.431130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766931.431130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766932.263134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766932.263165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766932.263214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766932.263231:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766932.263246:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263258:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766932.263267:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766932.263307:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076637824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766932.263367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263372:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766932.263375:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766932.263382:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263388:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766932.263396:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.263408:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766932.263417:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076637824:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766932.263432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076637952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766932.263453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766932.263459:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766932.263469:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263478:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766932.263485:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.263492:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775766932.263507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766932.263512:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076638080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766932.263526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766932.263532:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766932.263538:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263554:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766932.263556:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349640700 x1862023076637824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.263568:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.263578:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766932.263581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076637824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.263596:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.263601:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.263606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076637824:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766932.263613:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076638208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766932.263629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263633:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766932.263636:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766932.263643:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766932.263649:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263655:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766932.263660:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.263674:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766932.263694:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076638336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766932.263699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766932.263715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349640a80 x1862023076637952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.263716:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.263738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766932.263741:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766932.263744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076637952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.263755:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:2.0:1775766932.263762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766932.263765:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766932.263767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.263776:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076637952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766932.263783:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766932.263790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638080:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766932.263796:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775766932.263808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766932.263820:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076638464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766932.263823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349640e00 x1862023076638080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766932.263893:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076638080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.263924:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766932.263925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.263933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766932.263936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775766932.263939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766932.263942:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766932.263947:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638208:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766932.263954:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.263963:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766932.263969:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775766932.263974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766932.263990:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766932.263996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349641c00 x1862023076638208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766932.264017:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076638592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766932.264048:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076638208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.264059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264062:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264071:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638208:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766932.264084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766932.264088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766932.264091:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766932.264098:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:0.0:1775766932.264105:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766932.264112:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638336:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766932.264117:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.264127:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766932.264144:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076638720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766932.264156:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766932.264170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766932.264171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349641180 x1862023076638336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.264189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766932.264195:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766932.264213:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076638336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766932.264219:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3433 00000100:00080000:2.0:1775766932.264224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264230:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076638336:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766932.264238:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766932.264243:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766932.264252:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766932.264263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775766932.264271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766932.264283:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076638848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766932.264284:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349642680 x1862023076638464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766932.264323:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076638464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.264354:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264360:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264365:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766932.264374:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766932.264395:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766932.264403:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349640700 x1862023076638592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766932.264411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076638592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.264416:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264419:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264421:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766932.264435:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766932.264446:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766932.264452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349643800 x1862023076638720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766932.264460:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076638720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.264465:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264467:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264474:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766932.264479:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766932.264493:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766932.264499:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766932/real 1775766932] req@ffffa04349643b80 x1862023076638848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766932.264507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076638848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766932.264512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766932.264514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766932.264516:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076638848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766932.455179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766932.455179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766933.287238:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766933.287302:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766933.479267:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766933.479267:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766934.311153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766934.311167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766934.503171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766934.503172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766935.335192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766935.335207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766935.527126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766935.527126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775766936.359124:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766936.359205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775766936.551100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766936.551114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766937.383163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766937.383179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766937.383182:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766937.383201:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383213:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775766937.383218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766937.383224:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766937.383275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076638976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766937.383306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766937.383312:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766937.383316:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383322:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766937.383327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383336:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766937.383348:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076638976:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766937.383373:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076639104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766937.383385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383388:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766937.383390:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766937.383394:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383400:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766937.383405:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383411:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766937.383424:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076639232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766937.383434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383438:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766937.383443:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775766937.383444:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766937.383453:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383458:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766937.383463:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766937.383487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076639360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766937.383495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349643100 x1862023076638976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766937.383497:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383511:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766937.383519:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766937.383532:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383536:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766937.383536:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076638976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766937.383547:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383561:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775766937.383565:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.383571:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766937.383573:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076639488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766937.383582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383593:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766937.383595:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766937.383600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076638976:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766937.383605:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383616:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766937.383620:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383625:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766937.383631:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076639616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766937.383649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766937.383674:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766937.383680:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766937.383685:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383690:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775766937.383695:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766937.383696:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383708:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766937.383719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349642a00 x1862023076639104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766937.383721:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076639744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766937.383737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076639104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766937.383744:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766937.383764:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766937.383767:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766937.383768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.383789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766937.383790:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:0.0:1775766937.383805:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766937.383808:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766937.383815:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766937.383822:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766937.383823:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766937.383866:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766937.383871:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076639872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775766937.383878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349642300 x1862023076639232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766937.383925:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766937.383929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076639232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766937.383936:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766937.383944:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766937.383964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766937.383967:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3439 00000100:00080000:2.0:1775766937.383976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766937.383979:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766937.383981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639232:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766937.383987:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766937.383991:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639360:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775766937.383992:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766937.384015:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076640000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775766937.384047:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349640380 x1862023076639360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384079:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076639360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384085:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384087:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766937.384094:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766937.384109:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384115:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349641880 x1862023076639488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076639488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384128:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384132:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076639488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766937.384163:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766937.384176:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349641500 x1862023076639616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384191:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076639616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384202:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384204:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384206:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766937.384211:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775766937.384225:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384231:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349643100 x1862023076639744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384238:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076639744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384244:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384245:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384247:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766937.384251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766937.384261:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384267:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349643b80 x1862023076639872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384274:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076639872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384279:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384281:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076639872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766937.384304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775766937.384318:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766937.384325:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766937/real 1775766937] req@ffffa04349643800 x1862023076640000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766937.384333:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076640000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766937.384338:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766937.384340:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766937.384342:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766937.575132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766937.575139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766938.407127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766938.407232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766938.599194:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766938.599197:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766939.431192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766939.431232:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766939.623215:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766939.623215:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766940.455192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766940.455245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766940.647133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766940.647142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766941.479166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766941.479194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766941.671184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766941.671189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766942.503170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766942.503173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766942.503187:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503200:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00100000:1.0:1775766942.503204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766942.503213:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503221:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766942.503269:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076640128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766942.503295:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766942.503321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503326:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:2.0:1775766942.503330:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766942.503335:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076640128:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766942.503341:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766942.503357:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766942.503374:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076640256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766942.503384:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766942.503388:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503392:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:2.0:1775766942.503397:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503400:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766942.503403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766942.503410:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766942.503427:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076640384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766942.503436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766942.503441:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503445:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:2.0:1775766942.503450:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503454:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766942.503463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7e300 x1862023076640128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766942.503468:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766942.503487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076640512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766942.503493:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076640128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766942.503509:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766942.503532:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503536:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:2.0:1775766942.503539:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503543:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766942.503543:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766942.503555:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775766942.503556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.503563:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076640128:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766942.503577:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076640640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766942.503589:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766942.503591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503601:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766942.503604:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503610:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3444 00000800:00000100:3.0:1775766942.503617:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766942.503618:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503624:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766942.503630:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766942.503633:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7ed80 x1862023076640256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766942.503649:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076640768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766942.503654:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076640256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766942.503659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766942.503672:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503685:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:3.0:1775766942.503686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766942.503690:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766942.503695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766942.503701:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766942.503708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640256:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775766942.503714:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766942.503715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766942.503741:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076640896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766942.503754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766942.503758:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503763:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766942.503766:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766942.503772:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00100000:3.0:1775766942.503773:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7ce00 x1862023076640384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766942.503777:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503785:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766942.503794:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766942.503808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076640384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766942.503809:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076641024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766942.503874:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766942.503878:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766942.503879:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766942.503882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766942.503884:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3444 00000100:00080000:2.0:1775766942.503889:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766942.503894:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766942.503895:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775766942.503901:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766942.503906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766942.503922:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076641152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766942.503923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766942.503965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.503980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7df80 x1862023076640512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.503989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076640512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.503995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.503997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766942.504022:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766942.504087:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.504118:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7ea00 x1862023076640640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.504129:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076640640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.504153:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.504155:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504158:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076640640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766942.504162:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766942.504178:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.504185:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7d500 x1862023076640768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.504192:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076640768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.504197:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.504199:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766942.504205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766942.504214:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.504227:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7d880 x1862023076640896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.504235:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076640896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.504241:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.504243:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076640896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766942.504249:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766942.504260:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.504265:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7fb80 x1862023076641024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.504272:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076641024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.504277:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.504279:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504281:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766942.504284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766942.504294:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766942.504299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766942/real 1775766942] req@ffffa0434aa7e300 x1862023076641152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766942.504305:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076641152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766942.504315:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766942.504317:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766942.504319:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766942.695143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766942.695155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766943.527195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766943.527248:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766943.719130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766943.719130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766944.551119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766944.551198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766944.743131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766944.743131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766945.575138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766945.575167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766945.767132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766945.767132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766946.599133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766946.599175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766946.791132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766946.791132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775766947.623152:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766947.623163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766947.623164:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766947.623169:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623181:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623195:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623204:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775766947.623210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766947.623249:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076641280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766947.623294:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623297:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766947.623299:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623303:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623308:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623314:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623324:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766947.623336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076641280:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766947.623341:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076641408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766947.623353:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623356:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766947.623358:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623362:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623367:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623372:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623379:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766947.623395:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076641536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766947.623406:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766947.623418:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623423:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766947.623426:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623434:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623449:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766947.623449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350932300 x1862023076641280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766947.623461:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623473:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766947.623479:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076641280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766947.623488:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076641664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766947.623495:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766947.623501:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623505:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766947.623507:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775766947.623508:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623518:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00100000:3.0:1775766947.623521:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076641280:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775766947.623523:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623531:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623537:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775766947.623546:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076641792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766947.623554:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623557:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766947.623560:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766947.623563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641408:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766947.623567:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623574:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623579:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623583:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766947.623613:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076641920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766947.623639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766947.623648:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766947.623659:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350930700 x1862023076641408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766947.623694:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766947.623700:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766947.623706:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076641408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766947.623716:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623729:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623755:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623766:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766947.623767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.623776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.623780:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766947.623787:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076642048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766947.623790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766947.623806:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766947.623809:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766947.623812:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623817:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:1.0:1775766947.623823:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766947.623828:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766947.623832:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766947.623838:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766947.623846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350931c00 x1862023076641536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766947.623852:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076642176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766947.623870:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766947.623872:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076641536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766947.623876:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766947.623881:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766947.623892:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3449 00000100:00080000:3.0:1775766947.623895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.623900:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.623906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766947.623907:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766947.623917:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766947.623919:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641664:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775766947.623928:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766947.623944:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775766947.623953:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076642304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766947.623956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350932a00 x1862023076641664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.623993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076641664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624004:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766947.624026:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766947.624103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766947.624119:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350932680 x1862023076641792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.624136:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076641792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624145:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624148:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624151:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076641792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766947.624168:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766947.624187:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766947.624197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350931880 x1862023076641920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.624209:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076641920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624220:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624225:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624229:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076641920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766947.624237:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766947.624262:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766947.624287:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350930a80 x1862023076642048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.624301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076642048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624311:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624314:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766947.624319:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766947.624337:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766947.624346:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350930000 x1862023076642176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.624355:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076642176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624363:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624376:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766947.624386:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766947.624401:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766947.624409:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766947/real 1775766947] req@ffffa04350930e00 x1862023076642304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766947.624418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076642304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766947.624432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766947.624435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766947.624438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076642304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766947.815134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766947.815136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766948.647147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766948.647178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766948.839173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766948.839173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766949.671108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766949.671120:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766949.863096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766949.863104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766950.695112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766950.695156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766950.887101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766950.887101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766951.719117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766951.719127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766951.911101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766951.911124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766952.743092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775766952.743103:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743110:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766952.743113:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743122:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743131:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775766952.743136:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766952.743137:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775766952.743173:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076642432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766952.743194:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743197:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766952.743199:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743202:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743205:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766952.743208:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743215:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766952.743224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076642432:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766952.743241:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076642560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766952.743250:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743253:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766952.743255:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743258:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743261:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766952.743264:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743267:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766952.743276:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076642688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766952.743282:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766952.743290:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743293:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766952.743295:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743298:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743301:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766952.743303:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743307:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766952.743315:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076642816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766952.743319:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350930e00 x1862023076642432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766952.743331:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743339:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766952.743342:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766952.743349:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076642432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766952.743352:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743358:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766952.743364:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766952.743365:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766952.743369:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775766952.743372:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743376:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076642432:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766952.743381:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076642944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766952.743388:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743390:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766952.743391:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743394:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743396:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766952.743398:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766952.743402:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743406:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766952.743413:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076643072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766952.743420:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743423:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766952.743424:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743427:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743430:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766952.743433:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766952.743436:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743441:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766952.743449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350930000 x1862023076642560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766952.743452:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076643200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766952.743463:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076642560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766952.743466:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743468:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766952.743469:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743477:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743483:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766952.743487:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775766952.743487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766952.743494:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766952.743495:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743501:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775766952.743508:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076643328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766952.743509:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775766952.743521:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766952.743523:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766952.743525:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766952.743529:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3454 00000100:00080000:1.0:1775766952.743532:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766952.743536:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775766952.743538:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775766952.743543:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775766952.743550:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076643456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766952.743558:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350930a80 x1862023076642688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076642688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766952.743582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766952.743595:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743601:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350931880 x1862023076642816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076642816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743613:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743614:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743621:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766952.743625:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766952.743637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743644:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350932680 x1862023076642944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743651:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076642944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743660:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076642944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766952.743679:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766952.743691:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350932a00 x1862023076643072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743704:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076643072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743709:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743711:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766952.743716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766952.743726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350931c00 x1862023076643200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076643200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766952.743770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766952.743782:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743787:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350930700 x1862023076643328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076643328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743799:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743801:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743803:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766952.743806:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766952.743814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766952.743819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766952/real 1775766952] req@ffffa04350932300 x1862023076643456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766952.743826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076643456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766952.743835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766952.743837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766952.743839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076643456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766952.935158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766952.935158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775766953.767168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766953.767186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766953.959160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766953.959174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766954.791112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766954.791178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766954.983131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766954.983133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766955.815085:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766955.815162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766956.007112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766956.007141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766956.839126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766956.839202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766957.031146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766957.031177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766957.863154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766957.863174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766957.863203:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863207:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766957.863211:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863218:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863229:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863252:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766957.863286:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076643584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863315:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863320:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766957.863324:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863330:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863337:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863342:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863353:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766957.863355:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076643584:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766957.863369:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076643712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863378:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863381:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766957.863382:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863385:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863388:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863391:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863395:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766957.863403:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076643840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863409:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863412:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766957.863420:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863426:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3459 00000800:00000100:3.0:1775766957.863427:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766957.863430:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863434:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863439:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766957.863448:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076643968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863455:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863457:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766957.863459:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863462:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863464:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863467:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863470:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766957.863475:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7f800 x1862023076643584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766957.863481:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076644096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863503:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863506:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766957.863508:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863511:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00100000:3.0:1775766957.863515:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076643584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766957.863516:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863523:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863529:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766957.863535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.863542:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766957.863544:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076644224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766957.863551:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076643584:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766957.863554:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863560:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766957.863562:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863566:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863569:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863571:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863574:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766957.863580:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076644352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766957.863585:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766957.863593:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766957.863596:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766957.863597:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766957.863601:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863603:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863606:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863610:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766957.863620:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766957.863624:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076644480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766957.863635:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766957.863638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7c000 x1862023076643712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766957.863642:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766957.863646:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766957.863659:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076643712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766957.863660:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3459 00000100:00080000:2.0:1775766957.863665:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766957.863672:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766957.863680:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766957.863687:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766957.863693:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076644608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766957.863695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.863704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766957.863715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766957.863745:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.863758:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7f100 x1862023076643840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.863771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076643840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.863780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.863783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.863786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766957.863792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766957.863810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.863822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7dc00 x1862023076643968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.863835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076643968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.863843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.863846:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.863856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076643968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766957.863863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766957.863886:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.863896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7f480 x1862023076644096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.863904:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076644096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.863910:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.863912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.863914:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766957.863918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766957.863947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.863956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7e680 x1862023076644224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.863964:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076644224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.863969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.863972:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.863974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766957.863987:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766957.863999:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.864012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7c380 x1862023076644352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.864020:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076644352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.864025:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.864047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.864049:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766957.864053:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766957.864066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.864073:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7c700 x1862023076644480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.864081:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076644480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.864086:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.864088:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.864090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766957.864093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766957.864102:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766957.864107:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766957/real 1775766957] req@ffffa0434aa7f800 x1862023076644608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766957.864113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076644608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766957.864123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766957.864125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766957.864127:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076644608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766958.055154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766958.055169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766958.887145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766958.887183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766959.079135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766959.079136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766959.911124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766959.911162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766960.103146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766960.103146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766960.935142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766960.935183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766961.127120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766961.127121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766961.959134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766961.959187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766962.151103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766962.151149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766962.983165:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775766962.983166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766962.983177:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766962.983184:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766962.983184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766962.983197:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983209:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983216:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766962.983260:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076644736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983287:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983290:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766962.983291:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983294:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983298:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983301:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983310:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766962.983321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076644736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766962.983322:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076644864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983336:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983338:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766962.983358:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983362:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983365:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983368:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983373:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766962.983382:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076644992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983389:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983392:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766962.983393:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983396:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983398:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983400:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983403:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766962.983410:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766962.983414:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076645120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983421:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983424:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766962.983425:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983428:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983431:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983433:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983436:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766962.983449:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076645248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983456:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983459:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775766962.983461:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349640700 x1862023076644736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766962.983462:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983470:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983476:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983480:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983484:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766962.983487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076644736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766962.983494:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076645376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766962.983499:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766962.983504:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983506:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766962.983508:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983511:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983515:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766962.983518:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766962.983521:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766962.983525:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766962.983528:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076644736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766962.983536:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076645504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983542:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983545:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766962.983546:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766962.983555:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983559:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983563:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644864:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766962.983571:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766962.983583:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076645632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766962.983590:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766962.983593:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766962.983597:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766962.983598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766962.983604:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3464 00000100:00080000:0.0:1775766962.983609:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766962.983612:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349642680 x1862023076644864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766962.983618:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766962.983633:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076645760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766962.983634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076644864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983678:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983681:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766962.983689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766962.983706:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983713:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349641180 x1862023076644992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.983720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076644992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983726:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076644992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766962.983734:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076645120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766962.983744:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349641c00 x1862023076645120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.983757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076645120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983770:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076645120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766962.983774:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076645248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766962.983788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349640e00 x1862023076645248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.983802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076645248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076645248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766962.983819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766962.983837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349640a80 x1862023076645376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.983862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076645376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766962.983904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766962.983921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349643800 x1862023076645504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.983943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076645504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.983949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.983951:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.983953:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766962.983970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766962.983983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.983990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349643b80 x1862023076645632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.984000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076645632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.984009:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.984012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.984015:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766962.984021:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766962.984066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766962.984075:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766962/real 1775766962] req@ffffa04349643100 x1862023076645760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766962.984084:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076645760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766962.984094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766962.984097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766962.984099:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076645760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766963.175117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766963.175152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766964.007166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766964.007247:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766964.199144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766964.199182:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775766965.031162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766965.031239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766965.223097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766965.223107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766966.055128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766966.055185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766966.247154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766966.247155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766967.079174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766967.079226:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766967.271108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766967.271151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766968.103109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766968.103194:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103206:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766968.103213:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103225:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103238:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103266:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775766968.103303:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076645888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766968.103335:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103339:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766968.103341:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103347:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103352:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103356:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103366:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766968.103382:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076645888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766968.103385:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076646016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766968.103412:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103416:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766968.103417:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103425:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103429:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103432:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103437:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766968.103451:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076646144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775766968.103460:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766968.103461:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103469:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766968.103478:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103482:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103490:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103496:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103503:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766968.103517:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076646272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766968.103526:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103530:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766968.103532:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103537:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103542:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103546:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103553:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775766968.103565:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076646400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766968.103572:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646016:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766968.103589:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103593:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766968.103595:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103602:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103607:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103612:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103620:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766968.103622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775766968.103649:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076646528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766968.103659:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646272:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766968.103661:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103680:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766968.103683:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103690:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103696:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103702:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103709:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766968.103714:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766968.103751:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076646656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766968.103775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766968.103776:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103785:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766968.103789:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.103814:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00080000:2.0:1775766968.103861:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.103871:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.103881:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766968.103909:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076646784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766968.103919:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766968.103964:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766968.103971:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766968.103975:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766968.104002:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3469 00000100:00100000:0.0:1775766968.104006:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646784:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775766968.104015:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766968.104023:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766968.104117:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766968.104141:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076646912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766968.104226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766968.295215:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766968.295215:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775766969.127312:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766969.127340:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775766969.319155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766969.319156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775766970.151157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766970.151218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766970.343158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766970.343210:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766971.175104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766971.175135:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775766971.175311:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775766971.175354:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766971.175370:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175424:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175439:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175448:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175457:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175464:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175472:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175480:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775766971.175488:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775766971.175746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7f800 x1862023076646912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.175802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076646912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.175819:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.175826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.175832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766971.175843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7ed80 x1862023076646784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.175859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076646784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.175867:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.175872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.175876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766971.175916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7ce00 x1862023076646656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.175932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076646656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.175940:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.175944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.175947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766971.175955:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7ea00 x1862023076646400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.175967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076646400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.175976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.175979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.175983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766971.175991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7df80 x1862023076646528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.176004:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076646528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.176014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.176018:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.176022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076646528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766971.176076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7fb80 x1862023076646016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.176093:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076646016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.176103:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.176107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.176110:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766971.176118:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7d880 x1862023076646144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.176131:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076646144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.176139:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.176143:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.176146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766971.176155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7d500 x1862023076646272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.176168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076646272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.176177:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.176183:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.176190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076646272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766971.176207:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766968/real 1775766971] req@ffffa0434aa7e300 x1862023076645888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766971.176228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076645888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766971.176236:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766971.176241:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766971.176245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076645888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775766971.367146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766971.367160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766972.199119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766972.199198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766972.391117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766972.391169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775766973.223141:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766973.223147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766973.223149:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766973.223153:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223163:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223175:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223183:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775766973.223186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766973.223225:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076647040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223248:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223251:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766973.223252:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223255:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223258:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223275:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766973.223279:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076647040:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775766973.223286:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766973.223299:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076647168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223307:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223310:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766973.223311:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223314:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223317:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223319:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223322:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766973.223330:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076647296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223336:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223340:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775766973.223342:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766973.223342:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223349:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223351:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223354:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223357:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766973.223363:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076647424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223369:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223375:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766973.223377:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223380:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223383:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223386:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223390:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766973.223394:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7c700 x1862023076647040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766973.223401:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076647552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223422:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223425:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766973.223427:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223430:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223432:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223436:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223438:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766973.223445:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076647680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766973.223446:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076647040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223465:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766973.223465:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223470:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766973.223472:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766973.223476:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775766973.223478:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223483:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766973.223487:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076647040:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766973.223487:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223497:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766973.223506:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076647808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766973.223514:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766973.223518:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647168:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766973.223519:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766973.223522:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223529:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00080000:2.0:1775766973.223532:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223539:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223543:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766973.223550:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076647936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766973.223559:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775766973.223562:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766973.223567:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766973.223569:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766973.223574:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3474 00000100:00100000:0.0:1775766973.223578:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7c380 x1862023076647168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766973.223585:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766973.223596:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766973.223603:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766973.223621:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076648064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766973.223622:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076647168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223652:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.223658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.223663:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766973.223671:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766973.223699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.223715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7e680 x1862023076647296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.223727:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076647296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.223738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.223741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766973.223748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766973.223780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.223793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7f480 x1862023076647424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.223806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076647424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223844:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.223850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.223854:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766973.223860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766973.223891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.223903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7dc00 x1862023076647552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.223916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076647552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.223928:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.223931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076647552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766973.223936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766973.223956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.223971:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7f100 x1862023076647680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.223982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076647680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.223990:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.223992:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.224004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766973.224060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775766973.224083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.224095:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7c000 x1862023076647808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.224106:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076647808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.224113:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.224116:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.224119:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766973.224125:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766973.224142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.224152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7c700 x1862023076647936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.224164:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076647936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.224170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.224174:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.224177:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076647936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766973.224182:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775766973.224198:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766973.224216:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766973/real 1775766973] req@ffffa0434aa7e300 x1862023076648064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766973.224234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076648064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766973.224242:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766973.224245:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766973.224248:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766973.415167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766973.415167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766974.247135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766974.247251:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766974.439132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766974.439156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766975.271127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766975.271202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766975.463104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766975.463135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766976.295145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766976.295277:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766976.487107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766976.487123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775766977.319122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766977.319166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766977.511091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766977.511101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766978.343183:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766978.343183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766978.343194:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766978.343201:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766978.343213:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3480 00010000:00010000:1.0:1775766978.343215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766978.343225:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343234:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766978.343286:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076648192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766978.343316:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343321:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766978.343323:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766978.343329:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343335:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766978.343346:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076648192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766978.343360:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766978.343372:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766978.343390:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076648320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766978.343420:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766978.343423:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343430:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766978.343434:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766978.343443:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343452:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343460:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766978.343467:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349641500 x1862023076648192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766978.343469:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766978.343500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076648192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766978.343503:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076648448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766978.343546:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343558:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766978.343561:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766978.343562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766978.343571:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:3.0:1775766978.343571:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.343578:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076648192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766978.343581:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343599:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766978.343606:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766978.343621:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076648576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766978.343621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766978.343641:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775766978.343650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766978.343666:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766978.343669:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766978.343675:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349641880 x1862023076648320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766978.343677:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343687:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343694:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766978.343699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076648320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766978.343702:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766978.343741:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076648704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766978.343754:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.343765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.343771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766978.343772:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343780:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775766978.343782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648448:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766978.343784:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766978.343792:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343798:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343803:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766978.343808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766978.343809:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766978.343818:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349640380 x1862023076648448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766978.343828:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076648832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766978.343839:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076648448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766978.343841:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343852:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766978.343854:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766978.343856:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766978.343859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.343867:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648448:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766978.343868:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343873:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343884:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766978.343899:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766978.343899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766978.343920:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076648960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766978.343931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766978.343940:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.343945:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766978.343948:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766978.343950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349642300 x1862023076648576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766978.343957:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.343966:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766978.343990:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766978.343995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076648576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766978.344001:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766978.344018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775766978.344031:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076649088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766978.344032:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766978.344070:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766978.344095:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766978.344101:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766978.344108:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766978.344112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349642a00 x1862023076648704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766978.344114:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766978.344143:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3480 00000100:00080000:0.0:1775766978.344149:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766978.344149:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076648704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766978.344158:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766978.344170:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775766978.344171:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344180:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.344185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076648704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775766978.344192:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076649216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766978.344196:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766978.344236:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766978.344248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349641500 x1862023076648832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766978.344263:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076648832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766978.344269:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344271:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.344274:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766978.344287:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766978.344301:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766978.344308:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349643100 x1862023076648960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766978.344316:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076648960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766978.344321:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344323:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.344325:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076648960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766978.344329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766978.344338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766978.344344:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349643b80 x1862023076649088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766978.344352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076649088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766978.344366:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344369:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.344371:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766978.344374:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766978.344387:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766978.344393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766978/real 1775766978] req@ffffa04349643800 x1862023076649216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766978.344400:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076649216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766978.344423:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766978.344426:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766978.344428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766978.535101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766978.535101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766979.367124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766979.367270:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766979.559147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766979.559156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766980.391131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766980.391178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766980.583096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766980.583127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766981.415137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766981.415225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766981.607130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766981.607145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775766982.439239:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766982.439248:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775766982.631165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766982.631166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775766983.463143:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463154:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766983.463160:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463174:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00100000:2.0:1775766983.463176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766983.463178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766983.463186:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463198:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766983.463244:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076649344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463269:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463272:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766983.463274:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463277:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463280:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463284:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463294:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766983.463305:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076649344:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766983.463308:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076649472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463319:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463322:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766983.463323:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463325:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463327:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463329:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463351:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766983.463360:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076649600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463367:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463370:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766983.463371:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463373:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463377:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766983.463381:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766983.463384:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463388:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766983.463396:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076649728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463403:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463406:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766983.463407:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463409:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463412:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463415:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463418:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766983.463425:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349640a80 x1862023076649344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766983.463428:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076649856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463450:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463453:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775766983.463458:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076649344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766983.463460:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463468:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463475:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766983.463478:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766983.463480:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463486:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766983.463486:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463492:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076649344:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766983.463498:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076649984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463513:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463516:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766983.463517:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463520:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463522:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463524:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463527:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766983.463533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775766983.463537:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076650112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463546:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463549:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766983.463551:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463554:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:0.0:1775766983.463558:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766983.463563:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775766983.463565:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775766983.463567:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766983.463583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349640e00 x1862023076649472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766983.463587:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076650240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766983.463617:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766983.463621:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766983.463624:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076649472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766983.463625:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766983.463636:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3485 00000100:00080000:3.0:1775766983.463637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766983.463642:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766983.463642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766983.463651:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463657:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649600:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766983.463674:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766983.463694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766983.463700:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076650368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766983.463713:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349641c00 x1862023076649600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076649600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766983.463739:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766983.463752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463759:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349641180 x1862023076649728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076649728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463783:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766983.463786:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766983.463798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463804:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349642680 x1862023076649856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076649856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076649856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766983.463824:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766983.463834:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349640700 x1862023076649984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076649984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463860:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463863:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076649984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766983.463881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766983.463895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349640a80 x1862023076650112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076650112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766983.463921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766983.463931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349643800 x1862023076650240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463944:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076650240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.463950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.463957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766983.463961:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766983.463973:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766983.463979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766983/real 1775766983] req@ffffa04349643b80 x1862023076650368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766983.463990:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076650368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766983.463999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766983.464002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766983.464005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076650368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766983.655108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766983.655179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766984.487169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766984.487231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766984.679125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766984.679141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766985.511111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766985.511214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766985.703135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766985.703159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766986.535138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766986.535175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766986.727111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766986.727125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766987.559102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766987.559216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766987.751179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766987.751180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775766988.583159:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766988.583159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766988.583171:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766988.583177:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583190:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583204:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583214:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775766988.583223:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766988.583267:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076650496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583301:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766988.583306:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766988.583308:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583335:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583341:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766988.583345:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076650496:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766988.583350:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583367:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766988.583385:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076650624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583399:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766988.583403:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766988.583405:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583410:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583415:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583420:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583426:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766988.583438:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766988.583443:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076650752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583455:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766988.583460:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766988.583463:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583467:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583472:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583477:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583484:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766988.583495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349643100 x1862023076650496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766988.583500:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076650880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583519:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766988.583522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076650496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766988.583526:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766988.583531:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583543:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:3.0:1775766988.583547:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766988.583549:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766988.583557:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766988.583558:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583568:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766988.583568:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076650496:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775766988.583590:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076651008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583603:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766988.583622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766988.583624:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766988.583628:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583635:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583640:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583645:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583651:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766988.583659:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775766988.583675:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076651136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766988.583688:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766988.583688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349641500 x1862023076650624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766988.583698:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766988.583702:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766988.583714:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00100000:3.0:1775766988.583718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076650624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.583733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.583737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766988.583744:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766988.583744:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766988.583754:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766988.583758:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650752:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775766988.583764:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766988.583784:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076651264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766988.583794:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766988.583804:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766988.583808:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766988.583811:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766988.583812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349642a00 x1862023076650752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766988.583819:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00080000:0.0:1775766988.583824:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583829:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583835:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766988.583852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076650752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766988.583856:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076651392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766988.583877:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766988.583881:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766988.583885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766988.583888:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766988.583893:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766988.583893:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766988.583905:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3490 00000100:00100000:3.0:1775766988.583908:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650880:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775766988.583912:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766988.583918:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766988.583924:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766988.583936:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076651520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766988.583940:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.583955:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349642300 x1862023076650880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.583989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076650880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584015:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076650880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766988.584060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766988.584082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.584093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349640380 x1862023076651008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.584104:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076651008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584113:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584116:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584119:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766988.584124:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766988.584139:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.584148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349641880 x1862023076651136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.584159:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076651136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584170:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584180:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766988.584186:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766988.584206:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.584216:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349643100 x1862023076651264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.584228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076651264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584236:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584240:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584243:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766988.584249:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766988.584270:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.584282:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349643b80 x1862023076651392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.584295:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076651392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584304:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584310:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766988.584316:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766988.584336:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766988.584356:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766988/real 1775766988] req@ffffa04349643800 x1862023076651520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766988.584369:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076651520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766988.584377:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766988.584380:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766988.584384:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076651520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766988.775160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766988.775180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766989.607154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775766989.607225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766989.799175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766989.799175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766990.631197:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766990.631240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766990.823173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766990.823173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766991.655156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766991.655206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766991.847198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766991.847218:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766992.679101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766992.679242:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766992.871098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766992.871139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766993.703162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775766993.703167:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703175:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766993.703180:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703192:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703205:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775766993.703213:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766993.703215:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775766993.703256:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076651648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766993.703281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703284:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766993.703285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703291:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766993.703294:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703302:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766993.703313:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076651776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766993.703322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703327:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766993.703327:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076651648:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766993.703333:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703351:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703355:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766993.703358:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703380:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766993.703399:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076651904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766993.703411:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703413:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766993.703415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703418:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703422:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766993.703424:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766993.703429:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766993.703448:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076652032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766993.703455:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703457:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766993.703459:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703464:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703468:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766993.703489:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775766993.703494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349640a80 x1862023076651648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766993.703497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775766993.703521:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076652160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766993.703531:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076651648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766993.703549:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703572:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766993.703575:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.703581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766993.703591:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766993.703591:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076651648:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775766993.703622:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703628:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766993.703633:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766993.703652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076652288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766993.703658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651776:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766993.703666:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703669:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766993.703670:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703675:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703679:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766993.703683:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703688:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766993.703700:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076652416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766993.703705:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775766993.703719:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703723:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766993.703725:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766993.703729:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703735:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766993.703736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349640700 x1862023076651776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766993.703747:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703756:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775766993.703788:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076651776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766993.703792:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076652544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766993.703830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766993.703832:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766993.703838:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766993.703838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775766993.703846:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766993.703851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651776:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766993.703858:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3495 00000100:00080000:0.0:1775766993.703865:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766993.703866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651904:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775766993.703874:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775766993.703878:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775766993.703891:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076652672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766993.703896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.703913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349642680 x1862023076651904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.703923:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076651904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.703930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.703934:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.703936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076651904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766993.703940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766993.703952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.703958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349641180 x1862023076652032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.703965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076652032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.703981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.703984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.703986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766993.703991:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766993.704005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.704012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349641c00 x1862023076652160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.704019:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076652160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.704047:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.704049:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.704051:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766993.704069:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766993.704084:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.704091:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349640e00 x1862023076652288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.704099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076652288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.704106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.704108:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.704116:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766993.704121:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766993.704136:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.704142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349640a80 x1862023076652416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.704150:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076652416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.704156:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.704158:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.704160:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766993.704164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766993.704173:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.704179:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349643800 x1862023076652544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.704185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076652544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.704191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.704193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.704195:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766993.704199:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766993.704208:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766993.704221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766993/real 1775766993] req@ffffa04349643b80 x1862023076652672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766993.704229:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076652672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766993.704234:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766993.704236:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766993.704238:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076652672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775766993.895156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766993.895159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766994.727104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775766994.727211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766994.919171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766994.919171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775766995.751135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766995.751173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766995.943120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766995.943214:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766996.775158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775766996.775399:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766996.967127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766996.967154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766997.799220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766997.799282:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766997.991122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766997.991152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766998.823114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775766998.823163:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823168:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766998.823171:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823182:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823193:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823199:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775766998.823204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766998.823233:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076652800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823254:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823257:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766998.823258:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823261:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823263:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823288:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766998.823288:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076652800:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775766998.823301:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076652928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823312:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766998.823314:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823316:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823319:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823322:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766998.823334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076653056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823342:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766998.823344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823346:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823348:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823350:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823353:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766998.823359:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766998.823362:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076653184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823369:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823372:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766998.823377:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823380:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823383:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823386:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823390:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775766998.823401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7d500 x1862023076652800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766998.823402:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076653312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823437:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766998.823443:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766998.823448:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076652800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766998.823451:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823466:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823472:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823480:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775766998.823481:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775766998.823492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076653440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823515:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823517:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766998.823518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823522:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823524:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766998.823527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076652800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775766998.823538:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823543:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766998.823558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076653568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766998.823562:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652928:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775766998.823574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823576:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766998.823578:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823581:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823588:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823591:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823594:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775766998.823595:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775766998.823606:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076653696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766998.823613:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766998.823617:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766998.823617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7d880 x1862023076652928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766998.823620:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766998.823626:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3500 00000100:00080000:2.0:1775766998.823630:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766998.823637:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775766998.823640:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775766998.823646:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076653824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766998.823689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076652928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823704:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076652928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766998.823715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766998.823737:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.823746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7fb80 x1862023076653056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.823754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076653056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766998.823768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766998.823779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.823784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7df80 x1862023076653184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.823791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076653184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766998.823818:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766998.823839:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.823850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7ea00 x1862023076653312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.823863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076653312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076653312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766998.823885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766998.823902:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.823910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7ce00 x1862023076653440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.823917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076653440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766998.823947:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775766998.823961:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.823967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7ed80 x1862023076653568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.823975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076653568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.823980:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.823982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.823984:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775766998.823987:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766998.823996:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.824001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7f800 x1862023076653696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.824008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076653696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.824013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.824015:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.824017:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775766998.824020:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775766998.824049:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775766998.824063:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766998/real 1775766998] req@ffffa0434aa7d500 x1862023076653824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766998.824071:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076653824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766998.824077:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766998.824079:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775766998.824081:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076653824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775766999.015089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775766999.015143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775766999.847128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775766999.847169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767000.039159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767000.039229:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767000.871151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767000.871241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767001.063112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767001.063133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767001.895121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767001.895148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767002.087165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767002.087206:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767002.919162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767002.919246:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767003.111142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767003.111148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767003.943202:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943218:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767003.943229:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943239:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943250:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943256:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767003.943263:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767003.943279:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767003.943295:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076653952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943324:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943328:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767003.943330:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943335:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943340:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943345:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767003.943382:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076653952:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767003.943382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076654080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943415:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767003.943425:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943467:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943472:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943477:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767003.943496:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076654208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943503:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943511:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775767003.943512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767003.943519:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943525:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943529:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943531:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943538:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767003.943560:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076654336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943568:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943571:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767003.943573:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767003.943576:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7e300 x1862023076653952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767003.943581:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943591:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943607:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767003.943616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076653952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.943638:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.943645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767003.943646:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076654464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767003.943652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076653952:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767003.943665:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943670:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767003.943672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943676:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943680:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943686:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943692:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767003.943702:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076654592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943710:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767003.943712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767003.943713:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767003.943717:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943724:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943728:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943731:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943735:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767003.943746:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076654720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943752:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767003.943755:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767003.943756:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943761:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943771:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767003.943774:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767003.943779:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943784:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767003.943797:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076654848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767003.943803:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767003.943806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7c700 x1862023076654080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767003.943807:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767003.943809:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767003.943820:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3505 00000100:00080000:2.0:1775767003.943831:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767003.943840:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767003.943844:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767003.943852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076654080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767003.943862:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076654976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767003.943902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.943910:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.943915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767003.943927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767003.943958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.943980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7c000 x1862023076654208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076654208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944021:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944076:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767003.944088:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767003.944115:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7f100 x1862023076654336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944150:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076654336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944159:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944163:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944167:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767003.944175:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767003.944197:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944206:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7dc00 x1862023076654464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944214:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076654464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944221:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944223:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944225:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076654464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767003.944244:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767003.944257:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944264:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7f480 x1862023076654592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944271:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076654592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944287:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944292:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767003.944296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767003.944310:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944316:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7e680 x1862023076654720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944324:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076654720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944329:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944331:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944333:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767003.944337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767003.944346:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7c380 x1862023076654848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944359:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076654848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944364:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944366:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944372:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767003.944377:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767003.944395:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767003.944406:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767003/real 1775767003] req@ffffa0434aa7e300 x1862023076654976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767003.944419:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076654976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767003.944429:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767003.944433:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767003.944436:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076654976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767004.135137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767004.135162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767004.967184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767004.967309:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767005.159171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767005.159172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767005.991264:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767005.991305:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767006.183120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767006.183129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767007.015102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767007.015207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767007.207132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767007.207132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767008.039220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767008.039304:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767008.231128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767008.231163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767009.063144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767009.063145:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767009.063157:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767009.063170:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3510 00010000:00010000:3.0:1775767009.063175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767009.063182:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063192:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767009.063224:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076655104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063248:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063250:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767009.063252:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767009.063270:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063275:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063278:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063283:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076655104:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767009.063286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767009.063298:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076655232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063306:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063309:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767009.063310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767009.063313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063315:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063317:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767009.063320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767009.063327:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076655360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063333:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767009.063337:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767009.063341:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767009.063343:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063349:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767009.063353:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767009.063360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076655488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063371:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063374:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767009.063375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767009.063378:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349643100 x1862023076655104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767009.063381:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063387:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063407:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767009.063412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767009.063417:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076655104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767009.063424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076655616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767009.063429:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767009.063435:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767009.063437:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767009.063440:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767009.063442:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767009.063443:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076655104:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767009.063448:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063451:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063455:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767009.063461:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767009.063470:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076655744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063478:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767009.063480:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655232:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767009.063484:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767009.063487:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767009.063490:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063500:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063504:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767009.063507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767009.063512:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767009.063522:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349641880 x1862023076655232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767009.063527:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076655872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063542:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767009.063544:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076655232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767009.063547:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767009.063551:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767009.063560:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:3.0:1775767009.063562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767009.063563:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063569:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767009.063571:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775767009.063574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767009.063575:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767009.063582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767009.063586:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076656000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767009.063594:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767009.063596:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767009.063598:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767009.063602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767009.063605:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3510 00000100:00080000:0.0:1775767009.063609:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767009.063613:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349640380 x1862023076655360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767009.063620:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767009.063640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076656128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767009.063640:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076655360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063685:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767009.063697:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767009.063719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349642300 x1862023076655488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076655488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063749:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767009.063754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767009.063767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063773:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349642a00 x1862023076655616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076655616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063785:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076655616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767009.063802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076655744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767009.063812:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063818:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349641500 x1862023076655744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076655744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063840:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076655744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767009.063848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076655872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767009.063861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349643100 x1862023076655872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076655872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063878:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076655872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767009.063885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767009.063894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349643b80 x1862023076656000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063906:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076656000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767009.063922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767009.063934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767009.063940:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767009/real 1775767009] req@ffffa04349643800 x1862023076656128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767009.063947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076656128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767009.063952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767009.063953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767009.063955:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767009.255148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767009.255188:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767010.087117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767010.087152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767010.279098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767010.279181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767011.111171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767011.111171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767011.303215:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767011.303216:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767012.135168:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767012.135178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767012.327159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767012.327172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767013.159228:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767013.159403:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767013.351157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767013.351195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767014.183208:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183224:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767014.183233:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767014.183259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767014.183261:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00100000:2.0:1775767014.183279:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767014.183283:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183298:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767014.183419:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076656256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767014.183464:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183469:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767014.183472:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183478:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183483:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183488:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.183503:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767014.183519:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076656384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767014.183562:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076656256:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767014.183568:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183577:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767014.183580:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183588:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183594:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183599:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.183607:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767014.183627:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076656512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767014.183639:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183644:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767014.183646:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183650:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183658:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183663:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.183669:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767014.183676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767014.183686:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076656640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767014.183701:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183706:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767014.183709:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183714:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183729:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183736:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.183745:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767014.183750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350932300 x1862023076656256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767014.183768:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076656768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767014.183782:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767014.183787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076656256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767014.183790:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767014.183797:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183813:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183821:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183830:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767014.183834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767014.183836:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767014.183855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767014.183860:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076656896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767014.183873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076656256:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767014.183879:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183883:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767014.183886:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183892:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183902:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.183911:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.183917:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767014.183932:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076657024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767014.183952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767014.183968:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767014.183972:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767014.183979:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.183990:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.183997:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767014.184002:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.184011:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767014.184073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767014.184081:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076657152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767014.184097:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767014.184103:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350930700 x1862023076656384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767014.184107:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767014.184113:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767014.184129:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3515 00000100:00080000:1.0:1775767014.184139:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767014.184147:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076656384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767014.184148:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767014.184163:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767014.184177:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184183:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184192:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767014.184206:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076657280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767014.184207:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767014.184258:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184275:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350931c00 x1862023076656512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184290:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076656512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184312:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184316:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184320:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767014.184327:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767014.184395:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184412:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350932a00 x1862023076656640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184443:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076656640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184454:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184458:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184462:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767014.184469:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767014.184496:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350932680 x1862023076656768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076656768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184536:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184539:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076656768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767014.184546:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767014.184565:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184576:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350931880 x1862023076656896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076656896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184616:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076656896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767014.184650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767014.184676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350930a80 x1862023076657024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184703:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076657024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184716:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184720:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767014.184726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767014.184743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350930000 x1862023076657152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076657152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184775:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184779:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767014.184801:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767014.184842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767014.184863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767014/real 1775767014] req@ffffa04350930e00 x1862023076657280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767014.184889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076657280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767014.184906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767014.184912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767014.184919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076657280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767014.375172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767014.375173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767015.207196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767015.207237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767015.399199:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767015.399268:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767016.231118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767016.231154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767016.423152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767016.423152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767017.255129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767017.255150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767017.447122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767017.447122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767018.279103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767018.279141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767018.471140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767018.471189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767019.303133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767019.303154:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303162:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767019.303167:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303181:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303197:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767019.303205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767019.303206:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767019.303275:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076657408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303312:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303316:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767019.303317:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303321:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303331:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303336:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303347:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767019.303360:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076657536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767019.303368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076657408:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767019.303371:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303378:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767019.303381:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303386:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303389:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303391:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303418:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767019.303431:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076657664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303440:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303443:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767019.303444:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303446:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303457:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303463:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303470:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767019.303499:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076657792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767019.303504:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767019.303515:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303519:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767019.303522:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303528:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303533:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303537:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303542:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767019.303552:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076657920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303560:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767019.303573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350930e00 x1862023076657408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767019.303574:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767019.303582:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303591:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303596:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303599:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303604:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767019.303608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076657408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767019.303615:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076658048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303642:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303645:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767019.303646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767019.303648:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303655:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:3.0:1775767019.303657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767019.303658:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303665:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767019.303669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076657408:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767019.303670:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767019.303681:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076658176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303687:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303689:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767019.303690:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303694:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303696:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303698:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303701:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767019.303709:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767019.303716:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076658304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767019.303725:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767019.303729:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767019.303730:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767019.303732:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3520 00000100:00080000:1.0:1775767019.303735:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767019.303738:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767019.303741:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767019.303750:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076658432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767019.303762:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.303781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350930000 x1862023076657536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.303806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076657536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.303817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.303821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.303825:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767019.303832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767019.303853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.303871:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350930a80 x1862023076657664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.303884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076657664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.303894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.303898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.303902:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767019.303923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767019.303948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.303961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350931880 x1862023076657792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.303991:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076657792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304006:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304010:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767019.304018:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767019.304045:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.304086:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350932680 x1862023076657920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.304102:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076657920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304113:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304122:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076657920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767019.304138:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767019.304164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.304177:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350932a00 x1862023076658048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.304193:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076658048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304218:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304227:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767019.304234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767019.304264:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.304277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350931c00 x1862023076658176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.304312:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076658176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304331:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304335:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304339:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767019.304363:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767019.304388:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.304401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350930700 x1862023076658304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.304415:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076658304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304429:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767019.304450:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767019.304478:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767019.304492:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767019/real 1775767019] req@ffffa04350932300 x1862023076658432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767019.304508:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076658432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767019.304518:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767019.304522:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767019.304526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076658432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767019.495140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767019.495143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767020.327087:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767020.327163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767020.519155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767020.519173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767021.351145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767021.351225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767021.543194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767021.543195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767022.375116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767022.375152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767022.567187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767022.567192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767023.399113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767023.399221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767023.591104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767023.591104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767024.423157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767024.423157:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423170:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767024.423174:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423185:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423197:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767024.423198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767024.423206:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767024.423279:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076658560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767024.423328:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423332:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767024.423334:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423340:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423346:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423351:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767024.423357:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076658560:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767024.423363:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767024.423382:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076658688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767024.423393:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423397:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767024.423399:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423404:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423409:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767024.423413:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767024.423418:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423424:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767024.423440:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076658816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767024.423451:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767024.423452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350932300 x1862023076658560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767024.423458:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767024.423462:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423472:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423485:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767024.423485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076658560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767024.423495:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767024.423503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767024.423505:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767024.423509:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.423514:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076658560:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767024.423523:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076658944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767024.423537:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423540:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767024.423543:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767024.423546:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767024.423551:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423556:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423560:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423566:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767024.423571:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767024.423584:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076659072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767024.423584:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350930700 x1862023076658688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767024.423614:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076658688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767024.423619:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767024.423621:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767024.423625:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767024.423627:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767024.423629:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767024.423632:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767024.423639:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00100000:3.0:1775767024.423640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767024.423648:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423654:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767024.423660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423663:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767024.423673:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350931c00 x1862023076658816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767024.423680:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076659200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767024.423695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076658816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.423721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.423727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767024.423728:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767024.423731:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767024.423737:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767024.423740:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767024.423741:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658944:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767024.423752:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423758:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423763:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767024.423769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423770:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767024.423787:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076659328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767024.423788:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350932a00 x1862023076658944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767024.423798:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423809:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767024.423812:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423819:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423823:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423828:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423833:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767024.423834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076658944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767024.423849:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076659456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767024.423850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.423858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.423862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076658944:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767024.423877:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767024.423882:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767024.423883:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659072:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767024.423886:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767024.423894:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3526 00000100:00080000:1.0:1775767024.423906:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767024.423912:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767024.423916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767024.423921:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767024.423930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350932680 x1862023076659072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767024.423938:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076659584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767024.423947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076659072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.423960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.423964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.423968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767024.423975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767024.423995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767024.424003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350931880 x1862023076659200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767024.424011:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076659200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.424044:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.424047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.424050:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767024.424054:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767024.424070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767024.424077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350930a80 x1862023076659328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767024.424085:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076659328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.424090:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.424093:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.424094:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767024.424098:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767024.424107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767024.424112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350930000 x1862023076659456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767024.424119:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076659456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.424123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.424125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.424131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767024.424143:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767024.424156:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767024.424162:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767024/real 1775767024] req@ffffa04350930e00 x1862023076659584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767024.424169:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076659584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767024.424174:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767024.424176:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767024.424178:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076659584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767024.615198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767024.615198:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767025.447115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767025.447158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767025.639109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767025.639109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767026.471092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767026.471137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767026.663143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767026.663143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767027.495115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767027.495147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767027.687091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767027.687093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767028.519143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767028.519178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767028.711153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767028.711153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767029.543200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767029.543200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767029.543208:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767029.543224:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767029.543231:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543240:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543249:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543256:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767029.543291:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076659712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767029.543316:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767029.543318:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767029.543320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543323:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543325:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543328:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543335:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767029.543359:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076659712:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767029.543366:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076659840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767029.543388:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767029.543392:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767029.543394:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543399:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543407:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543412:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767029.543416:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543420:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767029.543433:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076659968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767029.543441:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767029.543444:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767029.543446:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543449:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00100000:3.0:1775767029.543453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350930e00 x1862023076659712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767029.543455:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543463:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767029.543490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076659712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767029.543493:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076660096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767029.543506:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767029.543508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767029.543512:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767029.543513:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767029.543515:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767029.543524:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076659712:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767029.543528:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543533:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543536:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543550:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767029.543558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076660224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767029.543560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767029.543575:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767029.543578:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767029.543579:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543583:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543586:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767029.543590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767029.543592:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543597:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767029.543605:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350930000 x1862023076659840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767029.543609:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076660352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767029.543630:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767029.543633:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076659840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767029.543634:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767029.543637:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767029.543644:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543649:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767029.543650:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767029.543653:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767029.543657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767029.543658:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767029.543661:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767029.543668:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076660480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767029.543669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767029.543681:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767029.543684:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767029.543686:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767029.543689:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767029.543690:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00080000:1.0:1775767029.543696:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543699:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767029.543704:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767029.543705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350930a80 x1862023076659968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767029.543716:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076660608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767029.543718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076659968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767029.543737:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767029.543739:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767029.543740:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767029.543744:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767029.543745:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767029.543749:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3531 00000100:00100000:3.0:1775767029.543751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076659968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767029.543753:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767029.543758:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660096:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767029.543763:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767029.543783:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076660736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767029.543790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350931880 x1862023076660096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.543824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076660096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.543832:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.543835:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.543838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767029.543854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767029.543873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350932680 x1862023076660224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.543888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076660224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.543894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.543896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.543898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767029.543902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767029.543912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350932a00 x1862023076660352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.543924:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076660352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.543935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.543938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.543940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767029.543943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767029.543956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350931c00 x1862023076660480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.543968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076660480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.543973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.543975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.543976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767029.543980:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767029.543989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.543994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350930700 x1862023076660608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.544001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076660608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.544005:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.544007:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.544013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767029.544016:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767029.544049:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767029.544057:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767029/real 1775767029] req@ffffa04350932300 x1862023076660736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767029.544065:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076660736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767029.544070:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767029.544072:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767029.544074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076660736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767029.735117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767029.735119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767030.567152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767030.567195:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767030.759136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767030.759141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767031.591100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767031.591146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767031.783134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767031.783135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767032.615196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767032.615220:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767032.807177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767032.807181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767033.639104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767033.639246:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767033.831187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767033.831187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767034.663150:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663159:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767034.663164:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767034.663168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767034.663171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767034.663174:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663184:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663201:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767034.663237:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076660864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663259:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663262:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767034.663263:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663266:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767034.663292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076660864:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767034.663293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076660992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663306:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767034.663309:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663312:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663315:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663317:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663352:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767034.663372:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076661120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663382:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767034.663383:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663385:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663389:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663392:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663396:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767034.663405:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076661248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663415:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767034.663415:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767034.663422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767034.663424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663427:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663430:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663433:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767034.663446:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076661376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663453:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663456:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767034.663458:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767034.663463:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663469:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663473:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663476:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663481:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767034.663492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076661504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767034.663500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661248:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767034.663505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767034.663508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663511:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663515:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663517:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663520:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767034.663532:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767034.663537:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076661632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663549:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663553:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767034.663555:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663563:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00100000:3.0:1775767034.663563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661504:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767034.663572:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767034.663578:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663584:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767034.663596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767034.663603:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076661760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767034.663618:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767034.663622:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767034.663624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767034.663630:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3536 00000100:00080000:1.0:1775767034.663636:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767034.663639:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661760:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767034.663645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767034.663652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767034.663669:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076661888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767034.663721:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767034.855149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767034.855149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767035.687109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767035.687176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767035.879106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767035.879141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767036.711120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767036.711162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767036.903110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767036.903138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767037.735130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767037.735213:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775767037.735360:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775767037.735393:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775767037.735406:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767037.735453:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775767037.735469:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767037.735469:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350932300 x1862023076660864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775767037.735477:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767037.735487:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767037.735493:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767037.735502:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767037.735507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076660864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775767037.735532:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767037.735533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000800:00000100:0.0:1775767037.735545:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767037.735550:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076660864:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767037.735572:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350930e00 x1862023076661888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735587:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076661888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735594:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767037.735608:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350930000 x1862023076661760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735620:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076661760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735642:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735646:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735649:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767037.735656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350931880 x1862023076661504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076661504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735680:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767037.735684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350930a80 x1862023076661632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735690:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076661632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735700:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076661632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767037.735706:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350932a00 x1862023076661248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076661248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735738:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767037.735744:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350932680 x1862023076661376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076661376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767037.735783:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350930700 x1862023076660992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735793:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076660992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735802:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076660992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767037.735816:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767034/real 1775767037] req@ffffa04350931c00 x1862023076661120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767037.735828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076661120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767037.735844:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767037.735849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767037.735853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076661120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767037.927117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767037.927163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767038.759146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767038.759181:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767038.951179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767038.951200:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767039.783190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767039.783190:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783201:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767039.783206:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783217:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783232:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783240:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767039.783241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767039.783285:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076662016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783331:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783337:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767039.783338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783343:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783349:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767039.783376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076662016:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767039.783381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076662144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783390:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767039.783395:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783398:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783401:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783403:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783406:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767039.783423:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076662272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783436:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767039.783443:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783451:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783465:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767039.783469:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767039.783475:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767039.783503:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076662400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783514:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783518:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767039.783520:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783527:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783534:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783539:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767039.783550:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350931c00 x1862023076662016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767039.783564:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076662528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783575:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767039.783582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767039.783585:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076662016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767039.783592:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00080000:1.0:1775767039.783602:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783611:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767039.783611:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767039.783620:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767039.783623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.783633:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076662016:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767039.783640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076662656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783665:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783669:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767039.783672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783679:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00100000:3.0:1775767039.783682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767039.783686:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783693:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783701:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767039.783725:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076662784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767039.783726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767039.783751:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783756:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767039.783758:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767039.783763:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00100000:3.0:1775767039.783767:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350930700 x1862023076662144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767039.783770:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767039.783779:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783790:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767039.783794:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076662144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767039.783807:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076662912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767039.783828:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767039.783833:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767039.783835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767039.783836:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767039.783848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767039.783849:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3541 00000100:00100000:3.0:1775767039.783856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767039.783866:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767039.783869:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662272:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767039.783875:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767039.783884:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767039.783903:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076663040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767039.784995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350932680 x1862023076662272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785088:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076662272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767039.785189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767039.785224:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785234:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350932a00 x1862023076662400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785245:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076662400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785254:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785257:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785274:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767039.785281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767039.785305:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785317:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350930a80 x1862023076662528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785325:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076662528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785332:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785334:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785336:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076662528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767039.785341:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767039.785356:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785362:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350931880 x1862023076662656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785370:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076662656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785375:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785378:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785379:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767039.785405:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767039.785425:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785445:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350930000 x1862023076662784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785457:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076662784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785465:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785468:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785472:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767039.785478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767039.785498:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785507:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350930e00 x1862023076662912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785515:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076662912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785521:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785523:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785525:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076662912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767039.785529:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767039.785540:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767039.785545:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767039/real 1775767039] req@ffffa04350932300 x1862023076663040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767039.785552:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076663040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767039.785562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767039.785565:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767039.785567:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767039.975161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767039.975162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767040.807167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767040.807170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767040.999121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767040.999121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767041.831154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767041.831165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767042.023164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767042.023169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767042.855135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767042.855155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767043.047159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767043.047159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767043.879106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767043.879140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767044.071111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767044.071112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767044.903143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767044.903143:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775767044.903153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767044.903157:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903166:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903175:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903181:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767044.903215:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076663168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767044.903238:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903241:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767044.903243:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903249:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903252:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903260:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767044.903273:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076663296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767044.903274:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076663168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767044.903289:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903291:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767044.903292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903295:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903296:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903299:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903318:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767044.903330:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076663424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767044.903342:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903346:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767044.903350:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767044.903351:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903365:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903371:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767044.903393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7d500 x1862023076663168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767044.903394:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076663552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767044.903416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903421:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767044.903422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903426:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903429:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903434:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903435:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076663168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767044.903438:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767044.903450:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076663680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767044.903454:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767044.903462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767044.903464:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767044.903470:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767044.903472:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076663168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767044.903478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903482:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903489:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767044.903498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767044.903505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076663808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767044.903513:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903516:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767044.903518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903521:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903524:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767044.903527:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767044.903552:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767044.903566:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076663936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767044.903574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767044.903576:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767044.903578:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903581:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903584:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767044.903588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7f800 x1862023076663296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767044.903591:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903598:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767044.903606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076663296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767044.903618:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076664064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767044.903623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903632:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663296:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767044.903637:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767044.903642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767044.903643:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767044.903648:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767044.903655:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3546 00000100:00080000:2.0:1775767044.903664:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767044.903685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767044.903689:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767044.903697:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767044.903698:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7ed80 x1862023076663424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076663424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767044.903717:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076664192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767044.903720:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767044.903738:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767044.903755:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7ce00 x1862023076663552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076663552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767044.903793:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767044.903806:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903813:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7ea00 x1862023076663680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076663680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903827:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903828:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903830:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076663680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767044.903844:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767044.903856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7df80 x1862023076663808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076663808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767044.903889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767044.903903:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7fb80 x1862023076663936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076663936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076663936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767044.903943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767044.903954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.903960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7d880 x1862023076664064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.903968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076664064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.903974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.903975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.903977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767044.903981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767044.903991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767044.904004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767044/real 1775767044] req@ffffa0434aa7d500 x1862023076664192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767044.904012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076664192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767044.904057:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767044.904060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767044.904063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767045.095099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767045.095102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767045.927127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767045.927158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767046.119113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767046.119120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767046.951166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767046.951195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767047.143103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767047.143133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767047.975137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767047.975223:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767048.167099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767048.167103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767048.999208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767048.999252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767049.191114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767049.191114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767050.023131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767050.023142:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023150:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767050.023154:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023165:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023176:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767050.023177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767050.023185:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767050.023225:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076664320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767050.023260:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023264:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767050.023267:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767050.023292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076664320:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767050.023299:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023305:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767050.023310:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767050.023320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767050.023338:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076664448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767050.023347:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767050.023357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767050.023363:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023369:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023373:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767050.023377:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7e300 x1862023076664320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767050.023382:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767050.023396:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076664576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767050.023405:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076664320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767050.023410:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023418:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767050.023420:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767050.023423:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767050.023424:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023429:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767050.023434:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767050.023434:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767050.023437:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767050.023445:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076664704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767050.023449:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076664320:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767050.023463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023466:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767050.023467:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023471:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767050.023478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664448:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767050.023479:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767050.023486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767050.023497:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076664832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767050.023505:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023509:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1775767050.023510:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767050.023513:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023518:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023521:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767050.023524:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767050.023527:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767050.023535:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7c380 x1862023076664448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767050.023536:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076664960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767050.023567:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767050.023572:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767050.023573:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076664448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767050.023578:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:3.0:1775767050.023584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767050.023588:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767050.023590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767050.023595:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664448:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767050.023600:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767050.023604:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767050.023611:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076665088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767050.023619:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767050.023622:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767050.023623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767050.023625:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023630:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023633:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767050.023637:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7e680 x1862023076664576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767050.023638:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767050.023643:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767050.023653:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076665216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767050.023661:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767050.023661:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076664576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767050.023666:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767050.023668:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767050.023675:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3551 00000100:00080000:2.0:1775767050.023679:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767050.023684:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767050.023685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767050.023687:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767050.023693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767050.023697:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076665344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767050.023701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767050.023712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767050.023740:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7f480 x1862023076664704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.023786:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076664704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.023793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.023796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.023800:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767050.023816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767050.023837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023847:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7dc00 x1862023076664832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.023858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076664832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.023866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.023868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.023871:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076664832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767050.023876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767050.023893:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7f100 x1862023076664960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.023914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076664960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.023928:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.023931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.023934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076664960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767050.023940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767050.023961:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.023986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7c000 x1862023076665088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.023994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076665088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.024000:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.024002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.024004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767050.024008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767050.024021:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.024045:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7c700 x1862023076665216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.024054:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076665216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.024059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.024061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.024068:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767050.024073:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767050.024088:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767050.024095:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767050/real 1775767050] req@ffffa0434aa7e300 x1862023076665344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767050.024102:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076665344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767050.024107:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767050.024109:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767050.024111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076665344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767050.215094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767050.215095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767051.047113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767051.047149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767051.239114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767051.239114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767052.071133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767052.071177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767052.263170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767052.263170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767053.095136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767053.095155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767053.287154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767053.287154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767054.119178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767054.119228:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767054.311151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767054.311151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767055.143178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767055.143201:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143208:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767055.143215:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143228:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143241:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143248:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767055.143279:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767055.143298:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076665472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767055.143325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143329:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767055.143331:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143335:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143338:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143341:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143351:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767055.143360:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076665472:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767055.143371:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076665600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767055.143384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767055.143390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143416:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143427:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143431:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143437:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767055.143447:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076665728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767055.143455:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143459:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767055.143461:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767055.143464:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767055.143467:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143470:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143473:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143477:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767055.143485:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076665856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767055.143492:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767055.143495:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143498:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143500:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143502:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143506:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767055.143514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349640a80 x1862023076665472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767055.143516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076665984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767055.143535:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143538:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767055.143539:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767055.143542:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076665472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767055.143545:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143551:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143557:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767055.143559:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767055.143561:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767055.143570:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076666112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767055.143577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767055.143580:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767055.143583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076665472:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767055.143585:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767055.143588:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143593:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143595:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143598:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767055.143608:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076666240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767055.143614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665600:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767055.143620:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767055.143623:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767055.143623:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767055.143627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143637:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767055.143642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767055.143646:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143653:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767055.143656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349640e00 x1862023076665600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.143669:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076665600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767055.143673:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076666368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767055.143685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.143694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767055.143696:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767055.143698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665600:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767055.143706:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767055.143710:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767055.143710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767055.143719:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3556 00000100:00080000:0.0:1775767055.143723:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767055.143727:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767055.143732:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767055.143732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.143741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349641c00 x1862023076665728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767055.143747:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076666496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767055.143761:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076665728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.143767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.143770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.143773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767055.143778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767055.143799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.143806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349641180 x1862023076665856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.143814:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076665856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.143820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.143822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.143824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767055.143828:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767055.143846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.143853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349642680 x1862023076665984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.143860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076665984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.143869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.143872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.143875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076665984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767055.143881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767055.143901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.143924:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349640700 x1862023076666112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.143950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076666112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.143957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.143959:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.143961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767055.143978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767055.143993:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.144000:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349640a80 x1862023076666240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.144008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076666240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.144014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.144016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.144018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767055.144021:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767055.144050:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.144057:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349643800 x1862023076666368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.144065:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076666368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.144080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.144083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.144087:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767055.144093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767055.144116:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767055.144126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767055/real 1775767055] req@ffffa04349643b80 x1862023076666496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767055.144139:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076666496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767055.144148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767055.144150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767055.144153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076666496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767055.335106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767055.335155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767056.167147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767056.167179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767056.359110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767056.359119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767057.191162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767057.191249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767057.383151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767057.383151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767058.215119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767058.215158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767058.407110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767058.407110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767059.239112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767059.239176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767059.431097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767059.431098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767060.263118:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263129:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767060.263134:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263144:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00100000:1.0:1775767060.263149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767060.263153:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263164:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767060.263177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767060.263199:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076666624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263238:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263241:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767060.263243:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263249:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263253:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263261:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767060.263270:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076666752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263280:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767060.263281:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263284:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263286:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263287:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263290:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767060.263298:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076666624:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767060.263299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076666880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767060.263316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263318:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263326:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263329:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263333:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767060.263343:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076667008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263352:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767060.263354:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263356:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263359:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263361:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767060.263365:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767060.263376:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076667136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263383:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767060.263386:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263389:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263392:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263394:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263397:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767060.263402:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349643100 x1862023076666624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767060.263407:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076667264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767060.263432:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076666624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767060.263437:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263445:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767060.263447:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263455:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263459:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767060.263459:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767060.263463:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263467:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767060.263467:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263474:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076666624:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767060.263480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076667392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263491:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767060.263492:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076666752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767060.263493:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263500:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3561 00000100:00080000:0.0:1775767060.263502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263504:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263508:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767060.263517:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076667520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767060.263523:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767060.263526:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767060.263527:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767060.263529:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3561 00000800:00000100:3.0:1775767060.263532:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767060.263539:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767060.263544:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767060.263548:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767060.263549:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349641500 x1862023076666752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767060.263560:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076667648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767060.263563:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076666752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263582:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263585:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263588:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076666752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767060.263608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076666880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767060.263621:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263627:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349642a00 x1862023076666880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263635:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076666880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263649:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263651:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076666880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767060.263660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767060.263675:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263681:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349642300 x1862023076667008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076667008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767060.263702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767060.263711:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263716:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349640380 x1862023076667136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076667136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263730:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767060.263735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767060.263744:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349641880 x1862023076667264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076667264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767060.263774:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767060.263786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349643b80 x1862023076667392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263798:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076667392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767060.263816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767060.263835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349643800 x1862023076667520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076667520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767060.263903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767060.263919:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767060.263926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767060/real 1775767060] req@ffffa04349640a80 x1862023076667648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767060.263933:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076667648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767060.263938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767060.263940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767060.263942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076667648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767060.455148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767060.455166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775767061.287184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767061.287366:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767061.479110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767061.479110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767062.311140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767062.311226:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767062.503099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767062.503099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767063.335170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767063.335185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767063.527128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767063.527129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767064.359134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767064.359162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767064.551098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767064.551112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767065.383150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767065.383166:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383175:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767065.383181:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383208:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383215:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767065.383261:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076667776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775767065.383274:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767065.383293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383297:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767065.383299:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383304:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383308:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383313:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383324:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767065.383341:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076667904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767065.383346:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076667776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767065.383379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383382:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767065.383383:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383387:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383391:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383394:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767065.383408:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076668032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767065.383415:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383418:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767065.383419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383421:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383425:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775767065.383431:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767065.383432:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383440:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767065.383451:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076668160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767065.383459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767065.383464:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383466:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383477:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383481:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767065.383491:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349640700 x1862023076667776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767065.383494:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767065.383507:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076668288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767065.383515:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383518:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767065.383521:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767065.383525:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076667776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767065.383530:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383535:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383543:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383550:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767065.383566:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767065.383567:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076668416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767065.383574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383577:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767065.383578:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383581:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383592:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767065.383598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767065.383599:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383607:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767065.383614:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076667776:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767065.383616:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076668544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767065.383663:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667904:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767065.383670:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767065.383675:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767065.383678:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767065.383684:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383699:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775767065.383700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767065.383709:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383719:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767065.383723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349642680 x1862023076667904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767065.383740:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076668672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767065.383745:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076667904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.383776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767065.383779:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767065.383785:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767065.383788:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767065.383795:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767065.383796:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076667904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767065.383827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767065.383833:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3567 00000100:00080000:0.0:1775767065.383840:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767065.383846:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767065.383852:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767065.383861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767065.383869:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076668800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767065.383880:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349641180 x1862023076668032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.383898:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076668032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.383911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.383916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.383920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767065.383927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767065.383960:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.383972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349641c00 x1862023076668160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.383986:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076668160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.383994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.383998:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767065.384008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767065.384054:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.384067:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349640e00 x1862023076668288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.384080:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076668288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.384089:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.384092:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384095:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076668288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767065.384101:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767065.384121:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.384133:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349640a80 x1862023076668416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.384159:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076668416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.384168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.384172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767065.384211:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767065.384233:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.384244:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349643800 x1862023076668544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.384256:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076668544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.384263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.384266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767065.384276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767065.384291:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.384300:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349643b80 x1862023076668672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.384311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076668672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.384329:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.384333:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384336:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767065.384342:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767065.384360:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767065.384369:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767065/real 1775767065] req@ffffa04349641880 x1862023076668800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767065.384379:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076668800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767065.384387:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767065.384391:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767065.384393:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076668800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767065.575143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767065.575143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767066.407122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767066.407188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767066.599083:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767066.599096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767067.431115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767067.431131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767067.623114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767067.623114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767068.455101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767068.455185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767068.647127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767068.647127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767069.479187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767069.479257:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767069.671127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767069.671127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767070.503107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767070.503125:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503133:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775767070.503136:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767070.503138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503149:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503183:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767070.503190:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767070.503223:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076668928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767070.503252:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503257:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767070.503260:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503264:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503268:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767070.503273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503283:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767070.503300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076669056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767070.503300:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076668928:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767070.503317:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503321:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767070.503323:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503328:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503332:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767070.503337:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767070.503356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076669184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767070.503368:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767070.503369:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767070.503385:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503391:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503397:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767070.503402:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349640380 x1862023076668928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503407:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503418:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767070.503423:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076668928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767070.503433:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767070.503435:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076669312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767070.503438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076668928:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767070.503458:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767070.503464:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503470:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00100000:1.0:1775767070.503471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669056:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767070.503474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767070.503481:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503487:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775767070.503500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767070.503503:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076669440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767070.503517:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767070.503517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349642300 x1862023076669056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503527:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767070.503532:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503542:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503546:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767070.503546:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076669056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503555:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767070.503558:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503567:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669056:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767070.503572:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767070.503574:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767070.503593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076669568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767070.503599:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.503609:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349642a00 x1862023076669184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503611:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503620:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767070.503623:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767070.503639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076669184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503643:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:1.0:1775767070.503647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503654:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767070.503657:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767070.503660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669312:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767070.503665:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503672:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767070.503678:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.503689:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349641500 x1862023076669312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767070.503690:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076669696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767070.503706:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076669312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503718:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767070.503718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503724:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767070.503728:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775767070.503729:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669312:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767070.503733:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767070.503739:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503744:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767070.503762:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669440:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767070.503766:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503775:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767070.503791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767070.503793:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076669824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767070.503806:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767070.503809:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767070.503818:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767070.503818:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349643100 x1862023076669440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503829:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3572 00000100:00080000:0.0:1775767070.503836:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767070.503837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076669440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767070.503845:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767070.503852:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767070.503857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076669440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767070.503870:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076669952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767070.503872:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767070.503895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.503904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349640380 x1862023076669568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767070.503912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076669568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767070.503917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767070.503924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767070.503933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.503938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349641880 x1862023076669696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767070.503945:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076669696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767070.503949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.503951:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.503958:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767070.503970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767070.503983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.503990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349643b80 x1862023076669824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767070.503997:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076669824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767070.504002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.504004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.504006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767070.504009:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767070.504017:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767070.504022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767070/real 1775767070] req@ffffa04349643800 x1862023076669952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767070.504048:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076669952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767070.504054:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767070.504056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767070.504058:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076669952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767070.695101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767070.695101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767071.527093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767071.527162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767071.719122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767071.719122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767072.551103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767072.551126:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767072.743113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767072.743116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767073.575120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767073.575173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767073.767114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767073.767114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767074.599151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767074.599176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767074.791149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767074.791149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767075.623125:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767075.623126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767075.623134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767075.623138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623147:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623155:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623162:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767075.623192:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076670080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623213:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623216:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767075.623217:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623221:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623224:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623227:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623235:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1775767075.623242:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767075.623247:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076670208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623257:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767075.623264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076670080:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767075.623273:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623278:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623282:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623285:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623290:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767075.623301:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076670336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623309:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623312:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767075.623313:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623315:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623318:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623319:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623323:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767075.623327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767075.623335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076670464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623341:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767075.623344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623346:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623360:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623363:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767075.623384:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076670592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623390:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767075.623394:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623397:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623400:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623403:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623407:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767075.623414:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076670720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623420:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767075.623424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623427:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623430:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623432:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623435:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767075.623438:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350932300 x1862023076670080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767075.623461:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076670848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623474:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623477:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767075.623479:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623490:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623495:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623500:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623506:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767075.623535:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076670976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767075.623546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767075.623549:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767075.623551:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767075.623555:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3577 00000100:00080000:1.0:1775767075.623560:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767075.623563:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767075.623567:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767075.623575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076671104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767075.623638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076670080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.623672:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.623687:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.623692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076670080:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767075.623792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767075.623813:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.623823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350930e00 x1862023076670208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.623832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076670208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.623838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.623841:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.623848:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767075.623853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767075.623867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.623878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350930000 x1862023076670336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.623890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076670336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.623899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.623903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.623906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767075.623913:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767075.623934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.623943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350931880 x1862023076670464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.623959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076670464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.623964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.623966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.623968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767075.623973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767075.623985:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.623998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350930a80 x1862023076670592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.624005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076670592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.624012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.624014:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.624016:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076670592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767075.624019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767075.624056:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.624067:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350932a00 x1862023076670720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.624075:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076670720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.624081:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.624083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.624086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767075.624090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767075.624103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.624113:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350932680 x1862023076670848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.624124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076670848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.624139:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.624143:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.624146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767075.624153:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767075.624175:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.624186:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350930700 x1862023076670976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.624194:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076670976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.624200:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.624202:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.624204:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076670976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767075.624207:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076671104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767075.624219:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767075.624225:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767075/real 1775767075] req@ffffa04350931c00 x1862023076671104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767075.624232:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076671104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767075.624237:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767075.624238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767075.624247:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076671104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767075.815099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767075.815099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767076.647150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767076.647204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767076.839163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767076.839163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767077.671183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767077.671225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767077.863133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767077.863190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767078.695114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767078.695186:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767078.887100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767078.887100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767079.719130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767079.719224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767079.911142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767079.911142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767080.743101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767080.743114:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743122:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767080.743127:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743137:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3582 00010000:00010000:3.0:1775767080.743145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767080.743167:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743173:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767080.743203:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076671232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767080.743227:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743230:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767080.743231:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743235:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:1.0:1775767080.743238:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743241:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743249:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767080.743255:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076671232:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767080.743261:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076671360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767080.743269:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743271:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767080.743272:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743274:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:1.0:1775767080.743276:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743278:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743281:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767080.743289:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076671488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767080.743295:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743304:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767080.743306:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743309:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3582 00000800:00000100:3.0:1775767080.743314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767080.743316:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743320:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743325:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767080.743334:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076671616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767080.743341:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743343:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767080.743345:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743347:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:1.0:1775767080.743349:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767080.743351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350931c00 x1862023076671232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767080.743354:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743360:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767080.743372:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076671744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767080.743377:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076671232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767080.743389:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743394:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767080.743396:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767080.743396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767080.743403:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:3.0:1775767080.743404:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767080.743408:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767080.743411:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076671232:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767080.743414:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743422:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767080.743430:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076671872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767080.743434:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767080.743448:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743450:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767080.743452:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743455:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3582 00000800:00000100:3.0:1775767080.743458:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767080.743460:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743464:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743469:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767080.743472:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350930700 x1862023076671360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767080.743479:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076672000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767080.743485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076671360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767080.743491:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767080.743499:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767080.743501:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743504:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:1.0:1775767080.743506:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767080.743512:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743518:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767080.743524:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767080.743528:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076672128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767080.743531:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743538:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767080.743546:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767080.743549:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767080.743550:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767080.743558:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767080.743562:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3582 00000100:00080000:1.0:1775767080.743567:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767080.743572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767080.743575:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767080.743579:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767080.743584:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350932680 x1862023076671488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767080.743591:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076672256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767080.743600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076671488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767080.743617:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767080.743629:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743634:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350932a00 x1862023076671616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743642:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076671616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743663:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743665:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743673:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767080.743677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767080.743699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743706:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350930a80 x1862023076671744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076671744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743720:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076671744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767080.743735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076671872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767080.743746:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350931880 x1862023076671872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076671872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743767:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076671872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767080.743770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767080.743779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350930000 x1862023076672000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076672000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743813:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743816:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767080.743822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767080.743842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743854:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350930e00 x1862023076672128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076672128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767080.743887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767080.743904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767080.743912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767080/real 1775767080] req@ffffa04350932300 x1862023076672256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767080.743920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076672256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767080.743930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767080.743932:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767080.743934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076672256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767080.935160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767080.935161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767081.767115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767081.767149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767081.959112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767081.959115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767082.791166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767082.791194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767082.983130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767082.983143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767083.815133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767083.815191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767084.007131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767084.007162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767084.839089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767084.839146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767085.031137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767085.031138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767085.863095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767085.863116:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863123:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775767085.863124:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767085.863127:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863138:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863148:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863155:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767085.863194:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076672384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767085.863221:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863225:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767085.863227:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863234:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863239:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863244:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863247:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076672384:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767085.863265:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767085.863284:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076672512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767085.863296:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863300:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767085.863303:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767085.863308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767085.863309:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863316:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863321:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767085.863328:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767085.863340:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076672640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767085.863344:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7d500 x1862023076672384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767085.863366:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863370:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767085.863372:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076672384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767085.863373:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863381:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:3.0:1775767085.863385:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767085.863386:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767085.863392:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767085.863396:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863398:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076672384:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767085.863403:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767085.863418:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076672768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767085.863419:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672512:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767085.863436:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863439:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775767085.863442:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767085.863449:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767085.863453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7d880 x1862023076672512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767085.863456:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863461:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863469:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767085.863476:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767085.863478:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076672512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863492:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767085.863496:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767085.863504:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076672896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767085.863514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767085.863515:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863520:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767085.863523:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863529:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00100000:3.0:1775767085.863531:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7fb80 x1862023076672640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767085.863533:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863543:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863547:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076672640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767085.863549:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767085.863560:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863563:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767085.863570:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076673024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767085.863584:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767085.863585:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672768:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767085.863603:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767085.863606:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863612:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863623:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767085.863626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767085.863630:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767085.863636:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767085.863643:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7df80 x1862023076672768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767085.863660:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076673152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767085.863660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076672768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767085.863688:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767085.863698:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767085.863703:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767085.863705:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672768:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767085.863718:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00100000:3.0:1775767085.863718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672896:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767085.863727:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767085.863733:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767085.863739:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767085.863742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7ea00 x1862023076672896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767085.863761:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076673280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767085.863766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076672896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863786:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767085.863788:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767085.863790:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767085.863795:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767085.863796:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076672896:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767085.863800:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767085.863807:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3587 00000100:00080000:2.0:1775767085.863811:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767085.863811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767085.863819:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767085.863825:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767085.863830:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7ce00 x1862023076673024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767085.863843:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076673408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767085.863862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076673024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863875:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767085.863888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767085.863902:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863908:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7ed80 x1862023076673152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767085.863916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076673152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767085.863927:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767085.863937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7f800 x1862023076673280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767085.863949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076673280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.863955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.863957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767085.863966:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767085.863975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767085.863986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767085/real 1775767085] req@ffffa0434aa7ea00 x1862023076673408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767085.863993:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076673408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767085.863999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767085.864000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767085.864002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076673408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767086.055109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767086.055121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775767086.887139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767086.887193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767087.079189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767087.079189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767087.911109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767087.911161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767088.103116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767088.103117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767088.935160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767088.935241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767089.127149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767089.127149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767089.959187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767089.959250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767090.151135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767090.151137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767090.983160:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983170:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767090.983174:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983185:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00100000:2.0:1775767090.983192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767090.983194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767090.983196:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767090.983233:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767090.983265:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076673536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767090.983291:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983295:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767090.983307:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983310:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:0.0:1775767090.983320:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767090.983323:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983333:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767090.983340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076673536:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767090.983346:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076673664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767090.983356:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983359:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767090.983360:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983363:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:0.0:1775767090.983365:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767090.983368:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983372:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767090.983383:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076673792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767090.983390:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983393:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767090.983400:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983404:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3592 00000800:00000100:3.0:1775767090.983409:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767090.983412:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767090.983418:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983424:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767090.983434:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076673920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767090.983442:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983445:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767090.983446:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983452:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00100000:3.0:1775767090.983452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349640a80 x1862023076673536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767090.983455:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767090.983464:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983480:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767090.983487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076673536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767090.983497:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076674048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767090.983498:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.983506:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.983515:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076673536:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767090.983522:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983527:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767090.983530:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983553:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:0.0:1775767090.983559:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767090.983565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767090.983566:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983576:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767090.983608:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767090.983611:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076674176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767090.983623:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983626:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767090.983628:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767090.983632:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:0.0:1775767090.983636:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767090.983636:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349640e00 x1862023076673664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767090.983644:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983653:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767090.983661:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076673664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767090.983666:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076674304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767090.983682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767090.983684:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983690:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767090.983690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767090.983695:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767090.983698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767090.983703:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:0.0:1775767090.983707:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767090.983710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767090.983714:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767090.983719:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767090.983730:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076674432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767090.983738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767090.983742:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767090.983747:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767090.983750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349641c00 x1862023076673792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767090.983752:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767090.983761:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076673792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767090.983772:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3592 00000100:00080000:3.0:1775767090.983773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.983781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767090.983783:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767090.983785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767090.983795:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767090.983795:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673920:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767090.983808:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767090.983817:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767090.983827:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076674560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767090.983829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349641180 x1862023076673920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.983862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076673920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.983876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.983880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.983893:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076673920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767090.983899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767090.983919:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767090.983926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349642680 x1862023076674048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.983935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076674048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.983941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.983944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.983946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767090.983961:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767090.983974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767090.983980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349640700 x1862023076674176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.983994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076674176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.984008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.984012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.984023:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767090.984076:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767090.984098:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767090.984108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349640a80 x1862023076674304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.984117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076674304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.984123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.984126:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.984128:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767090.984133:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767090.984144:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767090.984150:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349643800 x1862023076674432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.984158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076674432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.984164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.984166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.984168:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767090.984172:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767090.984182:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767090.984196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767090/real 1775767090] req@ffffa04349643b80 x1862023076674560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767090.984205:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076674560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767090.984212:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767090.984214:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767090.984216:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076674560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767091.175116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767091.175122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767092.007121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767092.007203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767092.199144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767092.199144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767093.031186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767093.031264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767093.223124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767093.223124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767094.055108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767094.055150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767094.247137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767094.247137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767095.079130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767095.079206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767095.271114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767095.271114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767096.103163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767096.103219:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103230:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767096.103235:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775767096.103241:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767096.103247:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103273:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103280:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767096.103314:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076674688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767096.103343:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103347:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767096.103349:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767096.103354:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103359:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103363:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103374:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767096.103388:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076674688:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767096.103390:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076674816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767096.103413:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103416:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767096.103419:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767096.103426:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103430:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103434:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103440:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767096.103456:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076674944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767096.103466:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103479:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767096.103496:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767096.103505:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767096.103510:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103517:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103523:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103532:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767096.103548:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076675072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767096.103560:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103563:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767096.103565:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767096.103571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350932300 x1862023076674688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767096.103576:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103583:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103597:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103604:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767096.103610:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076674688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767096.103620:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076675200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767096.103623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.103632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.103643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076674688:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767096.103653:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103658:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767096.103660:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767096.103668:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103675:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767096.103679:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767096.103684:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103700:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767096.103708:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767096.103719:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076675328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767096.103738:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767096.103739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350930e00 x1862023076674816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767096.103752:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767096.103758:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767096.103765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076674816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767096.103768:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103782:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767096.103782:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.103792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767096.103795:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767096.103803:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674816:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767096.103808:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767096.103824:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076675456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767096.103827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767096.103850:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103855:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767096.103857:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767096.103863:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103868:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767096.103873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767096.103879:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103887:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767096.103890:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350930000 x1862023076674944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.103905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076674944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767096.103911:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076675584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767096.103918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.103929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.103945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076674944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767096.103950:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767096.103955:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767096.103960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675072:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767096.103967:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767096.103975:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3597 00000100:00080000:1.0:1775767096.103981:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767096.103986:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767096.103994:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767096.103997:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350931880 x1862023076675072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767096.104019:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076675712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767096.104078:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076675072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104100:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767096.104106:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767096.104130:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350930a80 x1862023076675200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.104154:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076675200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104168:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767096.104189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767096.104205:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104213:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350932a00 x1862023076675328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.104221:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076675328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104227:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104229:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767096.104241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767096.104255:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104261:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350932680 x1862023076675456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.104269:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076675456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104276:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104278:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767096.104282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767096.104292:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104298:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350930700 x1862023076675584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.104305:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076675584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104310:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104312:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104314:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767096.104317:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767096.104326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767096.104338:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767096/real 1775767096] req@ffffa04350931c00 x1862023076675712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767096.104346:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076675712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767096.104351:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767096.104354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767096.104357:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076675712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767096.295116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767096.295156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767097.127121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767097.127162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767097.319142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767097.319153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767098.151154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767098.151236:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767098.343120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767098.343149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767099.175116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767099.175172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767099.367112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767099.367112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767100.199114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767100.199180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767100.391104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767100.391104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767101.223142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767101.223151:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223162:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767101.223168:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223181:0:52564:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3602 00010000:00010000:3.0:1775767101.223191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767101.223194:0:52564:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223206:0:52564:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767101.223254:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076675840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223278:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223281:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767101.223283:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223286:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223289:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767101.223314:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076675840:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767101.223324:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223334:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767101.223346:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076675968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223355:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223358:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767101.223360:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223363:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223373:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223376:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223382:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767101.223391:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076676096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223398:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223401:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767101.223402:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223404:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223406:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223409:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223413:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767101.223441:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076676224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223450:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767101.223453:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767101.223461:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767101.223463:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223467:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223471:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223474:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223484:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767101.223497:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076676352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767101.223500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767101.223513:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223517:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767101.223519:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223523:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223526:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767101.223531:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676224:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767101.223532:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223540:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767101.223557:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076676480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223565:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767101.223568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676352:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767101.223570:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767101.223574:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767101.223578:0:52564:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223581:0:52564:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223586:0:52564:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223595:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767101.223595:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767101.223610:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076676608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223623:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223626:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767101.223628:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767101.223632:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767101.223633:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223639:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223644:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223652:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767101.223662:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076676736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767101.223670:0:52564:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767101.223673:0:52564:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767101.223675:0:52564:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767101.223678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676736:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767101.223681:0:52564:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3602 00000100:00080000:1.0:1775767101.223687:0:52564:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767101.223690:0:52564:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767101.223694:0:52564:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767101.223704:0:52564:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076676864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767101.223753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767101.415095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767101.415099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767102.247109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767102.247170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767102.439126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767102.439126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767103.271127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767103.271157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767103.463115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767103.463115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767104.295111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767104.295129:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775767104.295293:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775767104.295337:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767104.295354:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767104.295426:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775767104.295445:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767104.295448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350932300 x1862023076676864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767104.295456:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767104.295468:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767104.295490:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767104.295496:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076676864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767104.295505:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767104.295517:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1775767104.295518:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767104.295528:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000800:00000100:1.0:1775767104.295534:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767104.295536:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767104.295548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350930e00 x1862023076676736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076676736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295575:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767104.295590:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350930000 x1862023076676608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295619:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076676608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295632:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.295638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350931880 x1862023076676480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076676480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295656:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295659:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076676480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.295668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350932a00 x1862023076676224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076676224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295687:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295692:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767104.295702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350930a80 x1862023076676352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295722:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076676352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295735:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767104.295745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350930700 x1862023076675968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076675968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295776:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076675968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.295784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350932680 x1862023076676096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076676096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076676096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.295817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767101/real 1775767104] req@ffffa04350931c00 x1862023076675840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.295844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076675840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.295852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.295856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767104.295858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076675840:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767104.487128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767104.487128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767105.319116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767105.319172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767105.511169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767105.511169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767106.343094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767106.343098:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343104:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767106.343108:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343117:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343127:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767106.343128:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767106.343134:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767106.343166:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076676992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767106.343191:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343194:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767106.343196:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343199:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343219:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343225:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343235:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767106.343240:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076676992:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767106.343256:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076677120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767106.343269:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343273:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767106.343275:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343279:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343284:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343287:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343292:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767106.343308:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076677248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767106.343309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767106.343322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343326:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767106.343328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343332:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343336:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343341:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343347:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767106.343353:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350931c00 x1862023076676992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767106.343365:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076677376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767106.343391:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076676992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767106.343422:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767106.343423:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.343429:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767106.343433:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767106.343437:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076676992:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767106.343438:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343447:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343453:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343458:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767106.343470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767106.343480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076677504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767106.343493:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343497:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767106.343500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767106.343502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767106.343507:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343514:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343522:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767106.343522:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350932680 x1862023076677120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767106.343530:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767106.343546:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076677632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767106.343564:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767106.343566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076677120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767106.343570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767106.343574:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343585:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:3.0:1775767106.343589:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.343594:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.343601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767106.343602:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343613:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767106.343617:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677248:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767106.343622:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767106.343640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076677760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767106.343651:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767106.343652:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343657:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767106.343660:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343667:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00100000:3.0:1775767106.343669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350930700 x1862023076677248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767106.343675:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343685:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343691:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767106.343692:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076677248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.343705:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767106.343710:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076677888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767106.343714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.343721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767106.343735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677376:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767106.343742:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767106.343746:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767106.343748:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767106.343753:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3608 00000100:00080000:1.0:1775767106.343758:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767106.343763:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767106.343768:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767106.343772:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767106.343783:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350930a80 x1862023076677376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767106.343788:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076678016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767106.343812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076677376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.343857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.343861:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.343865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767106.343885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767106.343980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767106.343992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350932a00 x1862023076677504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767106.344005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076677504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.344013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.344016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.344018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076677504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767106.344024:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767106.344075:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767106.344087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350931880 x1862023076677632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767106.344099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076677632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.344109:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.344112:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.344124:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767106.344131:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767106.344155:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767106.344167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350930000 x1862023076677760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767106.344179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076677760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.344188:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.344192:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.344195:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767106.344202:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767106.344224:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767106.344236:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350930e00 x1862023076677888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767106.344249:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076677888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.344259:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.344263:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.344266:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076677888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767106.344272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767106.344293:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767106.344321:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767106/real 1775767106] req@ffffa04350932300 x1862023076678016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767106.344335:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076678016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767106.344344:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767106.344348:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767106.344352:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767106.535118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767106.535157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767107.367119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767107.367153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767107.559146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767107.559167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767108.391142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767108.391185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767108.583103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767108.583105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767109.415110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767109.415148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767109.607138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767109.607138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767110.439141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767110.439251:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767110.631124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767110.631131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767111.463188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767111.463198:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463204:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767111.463209:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767111.463212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767111.463220:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463233:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463242:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767111.463283:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076678144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463318:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767111.463321:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463340:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463349:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463361:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767111.463361:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076678144:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767111.463377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767111.463399:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076678272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463414:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767111.463434:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463439:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463455:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463459:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767111.463467:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463469:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767111.463483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076678400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463494:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463496:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767111.463498:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463500:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463568:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463581:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767111.463593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076678528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463600:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463602:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767111.463604:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463607:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463610:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463615:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767111.463630:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076678656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463636:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463639:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767111.463640:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463642:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463645:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463647:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463650:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767111.463657:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076678784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463663:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463665:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767111.463666:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463668:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463671:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463674:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463677:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767111.463684:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076678912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463689:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463691:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767111.463692:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463702:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00080000:1.0:1775767111.463707:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463712:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767111.463712:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350932300 x1862023076678144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767111.463726:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767111.463739:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076679040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767111.463751:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767111.463781:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767111.463784:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767111.463793:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3613 00000100:00100000:3.0:1775767111.463800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076678144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767111.463801:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767111.463813:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767111.463826:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767111.463830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.463837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767111.463845:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076679168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767111.463845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076678144:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767111.463881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767111.463906:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.463917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350930e00 x1862023076678272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.463943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076678272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.463950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.463953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.463955:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767111.463960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767111.463975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.463982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350930000 x1862023076678400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.463990:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076678400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.463995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.463998:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767111.464003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767111.464014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350931880 x1862023076678528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464048:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076678528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464069:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464072:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767111.464083:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767111.464103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464111:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350932a00 x1862023076678656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464119:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076678656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464125:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464128:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464130:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076678656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767111.464133:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767111.464144:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464150:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350930a80 x1862023076678784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464157:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076678784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464165:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464171:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767111.464176:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767111.464190:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350930700 x1862023076678912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464204:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076678912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464209:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464211:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076678912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767111.464217:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767111.464227:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464233:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350932680 x1862023076679040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464240:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076679040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464245:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464247:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464249:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767111.464252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767111.464262:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767111.464273:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767111/real 1775767111] req@ffffa04350931c00 x1862023076679168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767111.464281:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076679168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767111.464286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767111.464288:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767111.464290:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767111.655137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767111.655166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767112.487181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767112.487262:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767112.679193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767112.679195:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767113.511181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767113.511264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767113.703119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767113.703147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767114.535143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767114.535202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767114.727115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767114.727115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767115.559143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767115.559197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767115.751096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767115.751111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767116.583105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767116.583136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583143:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767116.583147:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583157:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3618 00010000:00010000:0.0:1775767116.583158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767116.583169:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767116.583177:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767116.583213:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076679296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767116.583242:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767116.583249:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583253:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583258:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767116.583263:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767116.583284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076679296:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767116.583286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767116.583318:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076679424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767116.583330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583334:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767116.583336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583341:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583355:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767116.583366:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767116.583375:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767116.583397:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076679552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767116.583401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7df80 x1862023076679296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583424:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583430:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767116.583431:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076679296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583432:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:3.0:1775767116.583445:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767116.583449:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767116.583450:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.583457:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076679296:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767116.583459:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583468:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767116.583484:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076679680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767116.583484:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767116.583500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583504:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767116.583506:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583512:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583517:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767116.583525:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767116.583525:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583541:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767116.583550:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7fb80 x1862023076679424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767116.583557:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076679808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767116.583581:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767116.583587:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076679424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583597:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767116.583602:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767116.583617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767116.583618:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:3.0:1775767116.583626:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767116.583627:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767116.583634:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767116.583636:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583647:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767116.583648:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767116.583665:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076679936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767116.583677:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767116.583681:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767116.583684:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767116.583687:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583693:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583697:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767116.583699:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7d880 x1862023076679552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583703:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583721:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767116.583734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076679552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767116.583738:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076680064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767116.583759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767116.583765:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767116.583765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.583769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679552:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767116.583771:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767116.583776:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767116.583780:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583785:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767116.583788:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679680:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767116.583791:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583805:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767116.583813:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767116.583822:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076680192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767116.583829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7d500 x1862023076679680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583841:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767116.583845:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767116.583847:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767116.583850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076679680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767116.583855:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3618 00000100:00080000:2.0:1775767116.583863:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767116.583869:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767116.583880:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767116.583881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.583888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.583891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767116.583897:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076680320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767116.583898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767116.583923:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767116.583932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7e300 x1862023076679808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767116.583941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076679808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767116.583947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.583949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.583950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076679808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767116.583954:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767116.583964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767116.583970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7c700 x1862023076679936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767116.583977:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076679936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767116.583982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.583984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.583991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076679936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767116.583995:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767116.584007:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767116.584013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7c000 x1862023076680064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767116.584020:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076680064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767116.584053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.584056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.584059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767116.584076:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767116.584096:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767116.584107:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7f100 x1862023076680192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767116.584117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076680192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767116.584123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.584125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.584127:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767116.584131:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767116.584148:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767116.584166:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767116/real 1775767116] req@ffffa0434aa7dc00 x1862023076680320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767116.584179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076680320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767116.584187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767116.584191:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767116.584194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076680320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767116.775116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767116.775119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767117.607226:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767117.607280:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767117.799102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767117.799146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767118.631118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767118.631158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767118.823104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767118.823104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767119.655119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767119.655129:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767119.847091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767119.847093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767120.679113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767120.679198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767120.871107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767120.871111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767121.703092:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703101:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775767121.703101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767121.703106:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703115:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703124:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767121.703128:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767121.703132:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767121.703189:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076680448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767121.703213:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703228:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767121.703230:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703235:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703240:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767121.703246:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076680448:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767121.703256:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767121.703275:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076680576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767121.703284:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703287:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767121.703289:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703291:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703302:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767121.703307:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767121.703308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767121.703325:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076680704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767121.703332:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703334:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767121.703336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767121.703344:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349641880 x1862023076680448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767121.703348:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703355:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767121.703360:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703364:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767121.703365:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076680448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703375:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767121.703376:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076680832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767121.703383:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703391:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076680448:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767121.703392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767121.703398:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703402:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703404:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767121.703406:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703410:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767121.703414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767121.703418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076680960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767121.703430:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767121.703434:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767121.703437:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767121.703440:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703446:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00100000:3.0:1775767121.703447:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349640380 x1862023076680576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767121.703454:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767121.703460:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703468:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767121.703476:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076680576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767121.703483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076681088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767121.703484:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767121.703505:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767121.703508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703512:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767121.703513:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703517:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703523:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767121.703527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767121.703530:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703535:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767121.703538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349643100 x1862023076680704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767121.703548:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076681216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767121.703551:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076680704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767121.703571:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767121.703572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767121.703576:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767121.703579:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767121.703579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703585:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767121.703586:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703592:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767121.703596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680832:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767121.703597:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703605:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767121.703616:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076681344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767121.703622:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767121.703626:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775767121.703627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767121.703628:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767121.703633:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3623 00000100:00080000:0.0:1775767121.703641:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767121.703644:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349641500 x1862023076680832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767121.703647:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767121.703655:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767121.703666:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076681472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767121.703671:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076680832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703692:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767121.703710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767121.703725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349642a00 x1862023076680960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767121.703740:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076680960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703748:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703749:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076680960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767121.703753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767121.703761:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349642300 x1862023076681088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767121.703773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076681088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703780:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767121.703791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767121.703803:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703809:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349641880 x1862023076681216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767121.703816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076681216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767121.703827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767121.703837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703842:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349643b80 x1862023076681344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767121.703849:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076681344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703854:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767121.703860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767121.703868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767121.703881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767121/real 1775767121] req@ffffa04349643800 x1862023076681472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767121.703888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076681472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767121.703896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767121.703899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767121.703902:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076681472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767121.895135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767121.895146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767122.727161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767122.727280:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767122.919109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767122.919116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767123.751116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767123.751155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767123.943117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767123.943149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767124.775115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767124.775167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767124.967138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767124.967141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767125.799135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767125.799186:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767125.991129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767125.991129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767126.823141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767126.823243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823251:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767126.823256:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823265:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823274:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823279:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767126.823302:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767126.823316:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076681600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767126.823347:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823350:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767126.823352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823355:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823358:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823362:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823371:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767126.823383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076681728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767126.823384:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076681600:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767126.823419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767126.823424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823428:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823431:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823434:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823439:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767126.823450:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767126.823454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076681856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767126.823462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823465:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767126.823467:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823471:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823473:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823475:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823479:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767126.823488:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076681984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767126.823494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349640a80 x1862023076681600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767126.823501:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823509:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767126.823524:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823529:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823537:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767126.823541:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076681600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767126.823544:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767126.823565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076682112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767126.823566:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.823578:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767126.823579:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823583:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767126.823585:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823588:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823590:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823592:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823597:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767126.823600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076681600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767126.823632:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076682240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767126.823643:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823647:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767126.823649:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767126.823649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767126.823659:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823663:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823666:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823671:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767126.823683:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076682368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767126.823697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767126.823725:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823728:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767126.823730:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823734:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00080000:0.0:1775767126.823740:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823743:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767126.823749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349640700 x1862023076681728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767126.823762:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767126.823791:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076682496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767126.823817:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767126.823821:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767126.823823:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767126.823827:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3628 00000100:00100000:3.0:1775767126.823830:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076681728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767126.823831:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767126.823840:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767126.823855:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767126.823857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.823863:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767126.823868:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076682624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767126.823869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767126.823881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767126.823908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.823919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349642680 x1862023076681856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.823928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076681856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.823934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.823936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.823938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767126.823942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767126.823954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.823960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349641180 x1862023076681984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.823968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076681984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.823985:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.823988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.823990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076681984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767126.823994:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076682112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767126.824008:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.824015:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349641c00 x1862023076682112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.824024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076682112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.824046:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.824049:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.824051:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076682112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767126.824090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767126.824108:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.824126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349640e00 x1862023076682240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.824135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076682240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.824142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.824145:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.824153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767126.824158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767126.824199:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.824227:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349640a80 x1862023076682368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.824235:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076682368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.824241:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.824244:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.824246:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767126.824251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767126.824262:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.824268:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349643800 x1862023076682496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.824276:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076682496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.824281:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.824283:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.824285:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767126.824289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767126.824298:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767126.824312:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767126/real 1775767126] req@ffffa04349643b80 x1862023076682624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767126.824320:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076682624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767126.824326:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767126.824328:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767126.824331:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076682624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767127.015112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767127.015114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767127.847142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767127.847201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767128.039129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767128.039129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767128.871160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767128.871249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767129.063194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767129.063202:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767129.895140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767129.895177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767130.087154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767130.087165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767130.919169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767130.919201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767131.111112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767131.111132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767131.943140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767131.943167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767131.943196:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943202:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767131.943206:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943222:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943235:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943242:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767131.943283:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076682752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767131.943306:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943309:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767131.943310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943320:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767131.943342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076682880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767131.943343:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076682752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767131.943367:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767131.943372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943376:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943381:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943401:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767131.943419:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767131.943424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076683008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767131.943434:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943437:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767131.943439:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943442:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943446:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943449:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943454:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767131.943455:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7f480 x1862023076682752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767131.943466:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076683136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767131.943473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076682752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767131.943486:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943488:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767131.943489:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943493:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943497:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943503:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767131.943512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767131.943518:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076683264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767131.943520:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.943529:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076682752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767131.943533:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943546:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767131.943548:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943554:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00100000:3.0:1775767131.943557:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076682880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767131.943561:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943568:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767131.943585:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767131.943590:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076683392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767131.943598:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7e680 x1862023076682880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767131.943600:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943608:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767131.943612:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943615:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943619:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943625:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943632:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767131.943640:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076682880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767131.943643:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076683520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767131.943661:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767131.943662:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767131.943667:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767131.943669:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767131.943669:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767131.943677:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00100000:3.0:1775767131.943677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076682880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767131.943682:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767131.943688:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767131.943688:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683008:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767131.943699:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767131.943710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767131.943712:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076683648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767131.943720:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767131.943721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7c380 x1862023076683008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767131.943727:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767131.943730:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767131.943739:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3633 00000100:00080000:2.0:1775767131.943743:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767131.943745:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076683008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767131.943748:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767131.943755:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767131.943757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.943763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767131.943765:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076683776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767131.943769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767131.943776:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767131.943793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.943809:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7dc00 x1862023076683136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.943827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076683136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.943848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.943853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.943856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767131.943863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767131.943879:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.943887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7f100 x1862023076683264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.943894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076683264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.943900:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.943902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.943904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076683264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767131.943916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767131.943927:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.943932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7c000 x1862023076683392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.943939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076683392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.943945:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.943946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.943954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767131.943958:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767131.943970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.943976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7c700 x1862023076683520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.943984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076683520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.943989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.943991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.943993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767131.943996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767131.944005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.944010:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7e300 x1862023076683648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.944016:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076683648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.944022:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.944023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.944043:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767131.944047:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767131.944059:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767131.944071:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767131/real 1775767131] req@ffffa0434aa7d500 x1862023076683776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767131.944079:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076683776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767131.944085:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767131.944087:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767131.944088:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076683776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767132.135127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767132.135131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767132.967169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767132.967221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767133.159181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767133.159209:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767133.991190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767133.991238:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767134.183125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767134.183125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767135.015171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767135.015262:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767135.207160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767135.207161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767136.039168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767136.039253:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767136.231190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767136.231190:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767137.063189:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063206:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767137.063213:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767137.063219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767137.063252:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3638 00010000:00010000:1.0:1775767137.063265:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767137.063265:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063273:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767137.063313:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076683904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767137.063339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063357:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767137.063359:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063363:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063366:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063372:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063385:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767137.063433:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076684032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767137.063437:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076683904:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767137.063479:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063482:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767137.063483:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063487:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063491:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063495:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063500:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767137.063517:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076684160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767137.063545:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063551:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767137.063553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767137.063555:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063563:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063569:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063573:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063580:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767137.063593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076684288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767137.063601:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063608:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767137.063609:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7d880 x1862023076683904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767137.063611:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063624:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063632:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063636:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767137.063652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076684416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767137.063660:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063663:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767137.063665:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767137.063670:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076683904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767137.063674:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063692:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767137.063694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767137.063704:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767137.063705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767137.063716:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767137.063717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076683904:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767137.063739:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076684544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767137.063748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767137.063749:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063760:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767137.063766:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063770:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063775:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063778:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063782:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767137.063792:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076684672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767137.063802:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767137.063803:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063813:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767137.063815:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063820:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063824:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063828:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767137.063829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7fb80 x1862023076684032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767137.063833:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767137.063852:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076684800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767137.063861:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076684032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767137.063924:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767137.063926:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767137.063927:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767137.063931:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3638 00000100:00080000:2.0:1775767137.063934:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767137.063940:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767137.063947:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767137.063967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767137.063971:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076684928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767137.063980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.063991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767137.064009:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767137.064075:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064086:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7df80 x1862023076684160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064095:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076684160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064104:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064110:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767137.064117:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767137.064139:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064150:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7ea00 x1862023076684288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064161:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076684288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064177:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767137.064185:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767137.064209:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7f800 x1862023076684416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064233:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076684416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064243:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064247:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064250:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076684416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767137.064268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767137.064283:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064291:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7ed80 x1862023076684544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064298:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076684544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064304:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064306:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064308:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767137.064312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767137.064322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064335:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7ce00 x1862023076684672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064343:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076684672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064348:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064350:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767137.064356:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767137.064367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064373:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7d880 x1862023076684800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064380:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076684800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064385:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064387:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064388:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767137.064392:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767137.064400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767137.064405:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767137/real 1775767137] req@ffffa0434aa7d500 x1862023076684928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767137.064412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076684928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767137.064420:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767137.064423:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767137.064425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076684928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767137.255117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767137.255185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767138.087171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767138.087227:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767138.279144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767138.279144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767139.111226:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767139.111263:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767139.303188:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767139.303189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767140.135211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767140.135241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767140.327103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767140.327104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767141.159144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767141.159216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767141.351182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767141.351253:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767142.183141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767142.183146:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183152:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767142.183155:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183171:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183180:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183186:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767142.183205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767142.183219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076685056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183240:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767142.183244:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183250:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183253:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767142.183260:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767142.183271:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076685184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767142.183281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076685056:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767142.183286:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767142.183288:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183294:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183310:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183314:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767142.183324:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767142.183333:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076685312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183340:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767142.183344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767142.183349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767142.183351:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183355:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183357:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767142.183361:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767142.183372:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076685440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183383:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767142.183388:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350931c00 x1862023076685056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767142.183392:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183403:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183409:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767142.183411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076685056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767142.183416:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767142.183425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767142.183426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767142.183431:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183435:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076685056:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767142.183443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076685568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767142.183460:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767142.183481:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183485:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767142.183487:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183492:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3643 00000800:00000100:3.0:1775767142.183498:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767142.183500:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183507:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767142.183514:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767142.183516:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350932680 x1862023076685184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183527:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076685184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767142.183532:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076685696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183560:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767142.183560:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767142.183567:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767142.183570:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767142.183570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767142.183577:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00100000:3.0:1775767142.183580:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767142.183582:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183588:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685312:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767142.183594:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767142.183607:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076685824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767142.183614:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767142.183616:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767142.183617:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767142.183619:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3643 00000800:00000100:3.0:1775767142.183626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767142.183631:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767142.183637:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350930700 x1862023076685312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767142.183645:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767142.183664:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076685952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767142.183665:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076685312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767142.183689:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767142.183692:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767142.183693:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767142.183696:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767142.183701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767142.183702:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3643 00000100:00080000:1.0:1775767142.183709:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767142.183709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767142.183715:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685440:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767142.183721:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767142.183734:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076686080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767142.183743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183754:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350930a80 x1862023076685440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076685440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.183769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.183771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767142.183790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767142.183804:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350932a00 x1862023076685568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076685568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.183823:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.183825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076685568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767142.183831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767142.183840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183845:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350931880 x1862023076685696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076685696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.183857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.183859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767142.183864:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767142.183876:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350930000 x1862023076685824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076685824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.183916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.183919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767142.183928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767142.183945:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350930e00 x1862023076685952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076685952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.183966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.183968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.183970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076685952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767142.183973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767142.183982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767142.183987:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767142/real 1775767142] req@ffffa04350932300 x1862023076686080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767142.183994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076686080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767142.184061:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767142.184064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767142.184067:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767142.375132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767142.375134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767143.207102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767143.207171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767143.399117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767143.399117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767144.231110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767144.231151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767144.423100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767144.423111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767145.255173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767145.255215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767145.447116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767145.447116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767146.279096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767146.279143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767146.471096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767146.471096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767147.303162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767147.303211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767147.303224:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303237:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767147.303243:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303257:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303270:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303281:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767147.303329:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076686208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767147.303363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303371:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767147.303375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303383:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303390:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303397:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767147.303403:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076686208:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767147.303410:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767147.303436:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076686336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767147.303456:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767147.303485:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775767147.303489:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767147.303494:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303516:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303523:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767147.303534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767147.303557:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076686464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767147.303570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349641880 x1862023076686208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767147.303574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303581:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767147.303584:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303590:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303613:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767147.303617:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076686208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767147.303619:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767147.303626:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767147.303640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076686592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767147.303644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.303651:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767147.303656:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767147.303658:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076686208:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767147.303662:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767147.303664:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303670:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303675:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303680:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767147.303687:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767147.303690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767147.303711:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076686720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767147.303725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767147.303728:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303733:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767147.303736:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303742:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00100000:1.0:1775767147.303743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349642300 x1862023076686336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767147.303749:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303755:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767147.303763:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767147.303763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076686336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.303778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767147.303779:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076686848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767147.303784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.303790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767147.303795:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767147.303800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767147.303801:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767147.303804:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303809:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303815:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303822:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775767147.303829:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767147.303830:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767147.303845:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076686976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767147.303849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349642a00 x1862023076686464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.303870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076686464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767147.303873:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303879:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767147.303882:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767147.303885:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.303891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.303897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767147.303899:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00080000:0.0:1775767147.303905:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.303912:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767147.303916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686592:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767147.303921:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767147.303937:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076687104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767147.303980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767147.303983:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767147.303988:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767147.303991:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767147.303998:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3648 00000100:00100000:1.0:1775767147.304003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349641500 x1862023076686592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767147.304005:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767147.304011:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767147.304022:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767147.304060:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076686592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304076:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767147.304078:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076687232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767147.304083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767147.304143:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767147.304186:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767147.304204:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349643100 x1862023076686720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.304218:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076686720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304228:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.304231:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304235:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076686720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767147.304241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767147.304260:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767147.304270:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349640380 x1862023076686848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.304292:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076686848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304301:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.304304:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304307:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767147.304313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767147.304332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767147.304341:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349641880 x1862023076686976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.304353:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076686976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.304364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304367:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076686976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767147.304372:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076687104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767147.304387:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767147.304396:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349643b80 x1862023076687104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.304407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076687104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304430:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.304433:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304437:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076687104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767147.304443:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076687232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767147.304463:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767147.304472:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767147/real 1775767147] req@ffffa04349643800 x1862023076687232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767147.304484:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076687232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767147.304492:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767147.304495:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767147.304497:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076687232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767147.495162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767147.495165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767148.327135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767148.327237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767148.519128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767148.519128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767149.351107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767149.351137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767149.543121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767149.543122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767150.375101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767150.375179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767150.567100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767150.567103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767151.399088:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767151.399130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767151.591126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767151.591127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767152.423094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767152.423113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767152.423124:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423133:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423142:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423148:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767152.423154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767152.423182:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076687360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767152.423214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767152.423219:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423223:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423227:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423232:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767152.423241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076687360:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767152.423254:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076687488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767152.423262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767152.423266:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423269:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423272:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423275:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423278:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767152.423294:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076687616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767152.423296:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767152.423306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423308:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767152.423310:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423315:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423323:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423326:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423332:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767152.423333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349640a80 x1862023076687360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767152.423350:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076687744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767152.423365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423368:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767152.423369:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767152.423373:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076687360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767152.423374:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423379:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423385:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423392:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775767152.423393:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423400:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767152.423405:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076687872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767152.423406:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076687360:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767152.423418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767152.423422:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423425:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423428:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423432:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423432:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687488:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767152.423439:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767152.423450:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076688000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767152.423461:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423473:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349640e00 x1862023076687488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767152.423489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767152.423493:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076687488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767152.423494:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767152.423497:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423506:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:1.0:1775767152.423508:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767152.423514:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767152.423514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423520:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687488:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767152.423524:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423529:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687616:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767152.423534:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767152.423552:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076688128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767152.423554:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349641c00 x1862023076687616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767152.423582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767152.423588:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767152.423593:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00100000:1.0:1775767152.423594:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076687616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767152.423597:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423603:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423611:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767152.423622:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076688256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767152.423622:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423631:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687616:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767152.423639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767152.423646:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767152.423648:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767152.423650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687744:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767152.423654:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3654 00000100:00080000:0.0:1775767152.423665:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767152.423669:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767152.423675:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767152.423678:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767152.423689:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076688384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767152.423692:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349641180 x1862023076687744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.423717:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076687744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.423725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767152.423734:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767152.423745:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423751:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349642680 x1862023076687872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.423758:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076687872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.423763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076687872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767152.423785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767152.423798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349640700 x1862023076688000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.423832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076688000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.423841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767152.423855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767152.423875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423884:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349640a80 x1862023076688128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.423892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076688128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.423921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767152.423930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767152.423944:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423957:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349643800 x1862023076688256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.423965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076688256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.423971:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.423973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.423974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767152.423978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767152.423988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767152.423994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767152/real 1775767152] req@ffffa04349643b80 x1862023076688384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767152.424000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076688384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767152.424005:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767152.424007:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767152.424008:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076688384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767152.615140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767152.615140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767153.447135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767153.447163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767153.639109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767153.639109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767154.471157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767154.471157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767154.663134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767154.663134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767155.495160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767155.495297:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767155.687144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767155.687145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767156.519096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767156.519142:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767156.711098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767156.711121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767157.543141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767157.543154:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543162:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543177:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543183:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767157.543205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767157.543215:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076688512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767157.543241:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543244:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543248:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767157.543249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767157.543251:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767157.543269:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076688640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543293:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767157.543294:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767157.543300:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076688512:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767157.543301:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543306:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543311:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543315:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767157.543327:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076688768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543334:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767157.543338:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543340:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543354:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543357:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543360:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767157.543365:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767157.543369:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076688896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767157.543381:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543384:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543393:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543396:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767157.543407:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076689024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543417:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767157.543418:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543421:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543423:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543425:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543428:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767157.543437:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076689152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767157.543442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349641880 x1862023076688512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767157.543448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767157.543459:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543466:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543469:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543471:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543474:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767157.543480:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076689280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767157.543493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767157.543499:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767157.543500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076688512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767157.543504:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543509:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543515:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543523:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767157.543524:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.543531:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767157.543532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076689408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767157.543541:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076688512:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767157.543543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767157.543550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767157.543551:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767157.543577:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3659 00000100:00080000:0.0:1775767157.543581:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767157.543584:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767157.543588:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767157.543595:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767157.543599:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076689536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767157.543628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.543665:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349640380 x1862023076688640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.543682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076688640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.543691:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.543694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.543697:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767157.543704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767157.543723:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.543734:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349643100 x1862023076688768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.543747:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076688768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.543770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.543774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.543778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767157.543785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767157.543808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.543869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349641500 x1862023076688896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.543882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076688896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.543887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.543889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.543891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076688896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767157.543896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767157.543914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.543926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349642a00 x1862023076689024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.543938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076689024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.543947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.543950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.543961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767157.543968:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767157.543989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.544001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349642300 x1862023076689152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.544013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076689152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.544022:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.544046:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.544050:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767157.544057:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767157.544076:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.544087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349643b80 x1862023076689280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.544099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076689280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.544107:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.544110:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.544114:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767157.544120:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767157.544137:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.544155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349643800 x1862023076689408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.544167:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076689408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.544174:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.544177:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.544181:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767157.544206:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767157.544226:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767157.544236:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767157/real 1775767157] req@ffffa04349640a80 x1862023076689536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767157.544249:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076689536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767157.544257:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767157.544260:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767157.544263:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076689536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767157.735102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767157.735114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767158.567113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767158.567148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767158.759123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767158.759123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767159.591142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767159.591191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767159.783100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767159.783104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767160.615118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767160.615238:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767160.807110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767160.807110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767161.639138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767161.639356:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767161.831109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767161.831109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767162.663136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767162.663192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767162.663215:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663229:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3664 00010000:00010000:2.0:1775767162.663244:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767162.663267:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663276:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767162.663320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076689664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767162.663354:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663358:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663361:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663364:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663374:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767162.663385:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076689792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663393:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767162.663398:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663401:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663403:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663408:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663411:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767162.663414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076689664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767162.663419:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076689920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767162.663441:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663445:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663449:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663453:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767162.663468:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076690048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767162.663479:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663482:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663486:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663489:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663492:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775767162.663500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767162.663501:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076690176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663509:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767162.663514:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663518:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663528:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663531:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663542:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767162.663551:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349640700 x1862023076689664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767162.663555:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076690304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775767162.663589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076689664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767162.663590:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663602:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:1.0:1775767162.663606:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767162.663609:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767162.663613:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767162.663619:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767162.663619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076689664:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767162.663630:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767162.663633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767162.663646:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076690432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767162.663657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767162.663664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767162.663669:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775767162.663669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349642680 x1862023076689792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767162.663673:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663679:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663685:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663691:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767162.663701:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076689792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767162.663707:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076690560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767162.663750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767162.663751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767162.663753:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767162.663759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.663764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767162.663766:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767162.663776:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3664 00000100:00080000:0.0:1775767162.663782:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767162.663816:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767162.663823:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767162.663840:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076690688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767162.663840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767162.663876:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.663887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349641180 x1862023076689920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.663909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076689920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.663916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.663918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.663921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076689920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767162.663925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767162.663941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.663948:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349641c00 x1862023076690048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.663956:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076690048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.663961:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.663963:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.663965:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767162.663969:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767162.663981:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.663986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349640e00 x1862023076690176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.663994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076690176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.664010:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.664012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.664015:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767162.664019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767162.664034:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.664040:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349640a80 x1862023076690304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.664073:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076690304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.664079:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.664082:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.664084:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767162.664089:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767162.664103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.664109:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349643800 x1862023076690432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.664118:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076690432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.664123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.664125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.664132:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767162.664137:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767162.664152:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.664159:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349643b80 x1862023076690560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.664167:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076690560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.664173:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.664174:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.664176:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767162.664194:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767162.664205:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767162.664212:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767162/real 1775767162] req@ffffa04349642300 x1862023076690688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767162.664220:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076690688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767162.664226:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767162.664227:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767162.664229:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076690688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767162.855142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767162.855142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767163.687118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767163.687218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767163.879101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767163.879108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767164.711126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767164.711255:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767164.903209:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767164.903243:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767165.735127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767165.735248:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767165.927146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767165.927148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767166.759162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767166.759212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767166.951101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767166.951103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767167.783148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767167.783164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767167.783166:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783184:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783200:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783211:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767167.783265:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076690816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775767167.783288:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767167.783307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783314:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767167.783317:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783324:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783328:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783334:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783348:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767167.783377:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076690816:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767167.783393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076690944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767167.783405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767167.783411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783415:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783420:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783423:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783429:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767167.783444:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076691072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767167.783453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767167.783458:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783461:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783476:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783480:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783485:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767167.783495:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076691200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767167.783504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767167.783509:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783521:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783527:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783532:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783538:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767167.783552:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076691328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767167.783561:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767167.783567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767167.783577:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783583:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783586:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783591:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783597:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767167.783611:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076691456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767167.783627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767167.783628:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767167.783644:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783652:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783660:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783667:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783677:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767167.783677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767167.783707:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076691584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767167.783730:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691328:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767167.783768:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783772:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767167.783775:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783781:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783788:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783792:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783800:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767167.783807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767167.783818:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076691712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767167.783829:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767167.783832:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767167.783834:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767167.783838:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3669 00000100:00080000:0.0:1775767167.783842:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767167.783845:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767167.783850:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767167.783861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767167.783864:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076691840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767167.783911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767167.783941:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767167.975152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767167.975152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767168.807108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767168.807302:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767168.999135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767168.999136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767169.831128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767169.831246:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767170.023124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767170.023134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767170.855126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767170.855291:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775767170.855326:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775767170.855360:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767170.855373:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855432:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855446:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855456:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855463:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855471:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855479:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855489:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767170.855497:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775767170.855569:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349640a80 x1862023076691840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855594:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076691840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855612:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767170.855625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349643b80 x1862023076691584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076691584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855654:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767170.855680:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349643800 x1862023076691712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076691712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855719:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767170.855730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349641880 x1862023076691328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076691328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855753:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767170.855770:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349642300 x1862023076691456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076691456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855795:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855801:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076691456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767170.855815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349641500 x1862023076690944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076690944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855846:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076690944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767170.855853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349643100 x1862023076691072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076691072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855880:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767170.855886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349640380 x1862023076691200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855898:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076691200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855909:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855912:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076691200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767170.855925:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767167/real 1775767170] req@ffffa04349642a00 x1862023076690816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767170.855938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076690816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767170.855947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767170.855950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767170.855953:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076690816:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767171.047127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767171.047127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767171.879101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767171.879234:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767172.071118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767172.071118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767172.903109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767172.903124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767172.903136:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903147:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:0.0:1775767172.903172:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903180:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767172.903207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767172.903216:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076691968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767172.903262:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903266:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:0.0:1775767172.903271:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903276:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767172.903294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076691968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767172.903302:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076692096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767172.903332:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903336:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:0.0:1775767172.903340:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903344:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767172.903362:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076692224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903373:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775767172.903376:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767172.903383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767172.903386:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903392:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:0.0:1775767172.903398:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903403:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903410:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767172.903424:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349640e00 x1862023076691968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767172.903428:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076692352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903457:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767172.903465:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767172.903465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076691968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767172.903475:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:0.0:1775767172.903481:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903486:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767172.903487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767172.903493:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775767172.903496:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.903502:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076691968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767172.903511:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076692480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903522:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767172.903528:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903533:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00100000:1.0:1775767172.903536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767172.903539:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903547:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903560:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767172.903570:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767172.903577:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076692608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767172.903593:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767172.903599:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767172.903606:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00100000:1.0:1775767172.903606:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349641c00 x1862023076692096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767172.903613:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903622:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903631:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767172.903638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076692096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767172.903646:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076692736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767172.903651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.903662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767172.903666:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767172.903673:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767172.903676:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767172.903697:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00100000:1.0:1775767172.903698:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767172.903705:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767172.903711:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767172.903719:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767172.903728:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767172.903736:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076692864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767172.903743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349641180 x1862023076692224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767172.903755:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767172.903761:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775767172.903764:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076692224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767172.903771:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767172.903776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767172.903781:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3674 00000100:00080000:1.0:1775767172.903784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767172.903789:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767172.903789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767172.903799:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767172.903802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692352:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767172.903808:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767172.903826:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076692992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767172.903840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.903855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349642680 x1862023076692352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.903870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076692352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.903886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.903891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.903894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767172.903902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767172.903926:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.903936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349640700 x1862023076692480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.903947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076692480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.903956:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.903959:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.903962:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076692480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767172.903986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767172.904003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.904014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349640e00 x1862023076692608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.904061:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076692608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.904071:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.904074:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.904086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767172.904093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767172.904119:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.904131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349642a00 x1862023076692736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.904143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076692736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.904151:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.904154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.904158:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767172.904164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767172.904184:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.904196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349640380 x1862023076692864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.904207:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076692864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.904215:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.904218:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.904222:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767172.904228:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767172.904247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767172.904267:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767172/real 1775767172] req@ffffa04349643100 x1862023076692992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767172.904276:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076692992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767172.904281:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767172.904284:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767172.904288:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076692992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767173.095142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767173.095142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767173.927121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767173.927284:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767174.119097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767174.119120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767174.951109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767174.951148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767175.143085:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767175.143114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767175.975152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767175.975304:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767176.167117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767176.167117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767176.999125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767176.999146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767177.191111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767177.191111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767178.023129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767178.023133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767178.023142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767178.023146:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775767178.023157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767178.023157:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00080000:2.0:1775767178.023170:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023176:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767178.023215:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076693120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767178.023256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767178.023261:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023266:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00080000:2.0:1775767178.023271:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023276:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767178.023286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767178.023287:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076693120:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767178.023304:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076693248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767178.023316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023319:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767178.023321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00080000:2.0:1775767178.023329:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767178.023338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767178.023344:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767178.023352:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076693376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767178.023363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767178.023369:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023373:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00100000:1.0:1775767178.023379:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7e300 x1862023076693120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767178.023388:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023396:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076693120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767178.023406:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767178.023408:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023413:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767178.023423:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076693504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767178.023426:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076693120:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767178.023442:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767178.023446:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023450:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00100000:1.0:1775767178.023453:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767178.023456:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023461:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767178.023465:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775767178.023478:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767178.023479:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076693632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767178.023489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767178.023492:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7c700 x1862023076693248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767178.023493:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767178.023496:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023504:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00080000:2.0:1775767178.023510:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023514:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023517:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076693248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767178.023519:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767178.023528:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023530:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.023532:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767178.023538:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767178.023542:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076693760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767178.023557:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767178.023566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767178.023568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7c000 x1862023076693376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767178.023570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767178.023573:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023581:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00100000:1.0:1775767178.023586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076693376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767178.023586:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023594:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767178.023601:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767178.023603:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767178.023613:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076693888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767178.023618:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767178.023638:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767178.023643:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023646:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00100000:1.0:1775767178.023658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767178.023680:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767178.023681:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767178.023686:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767178.023703:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767178.023704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7f100 x1862023076693504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767178.023716:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076694016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767178.023724:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076693504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767178.023725:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767178.023732:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767178.023734:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767178.023743:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3679 00000100:00080000:1.0:1775767178.023747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767178.023753:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767178.023756:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767178.023760:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693504:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775767178.023768:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767178.023779:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076694144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767178.023784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767178.023809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7dc00 x1862023076693632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767178.023825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076693632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767178.023831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.023835:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076693632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767178.023839:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076693760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767178.023858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7c380 x1862023076693760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767178.023872:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076693760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767178.023883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.023888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076693760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767178.023892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076693888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767178.023904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7e680 x1862023076693888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767178.023917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076693888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767178.023923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.023927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076693888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767178.023930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767178.023945:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767178.023952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7f480 x1862023076694016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767178.023964:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076694016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767178.023972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.023975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.023984:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767178.024002:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767178.024022:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767178.024048:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767178/real 1775767178] req@ffffa0434aa7e300 x1862023076694144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767178.024056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076694144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767178.024062:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767178.024064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767178.024066:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767178.215102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767178.215103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767179.047123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767179.047277:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767179.239127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767179.239127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767180.071134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767180.071158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767180.263115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767180.263116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767181.095122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767181.095152:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767181.287105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767181.287107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767182.119093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767182.119158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767182.311124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767182.311124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767183.143129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767183.143139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767183.143139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767183.143145:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767183.143148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767183.143155:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:2.0:1775767183.143166:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143174:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767183.143213:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076694272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767183.143241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767183.143247:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767183.143253:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:2.0:1775767183.143258:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143262:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143269:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076694272:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767183.143272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767183.143290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076694400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767183.143316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143320:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767183.143322:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767183.143327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767183.143330:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:2.0:1775767183.143337:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143342:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767183.143349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767183.143363:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7d500 x1862023076694272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767183.143368:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076694528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767183.143385:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076694272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767183.143399:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767183.143402:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767183.143406:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:0.0:1775767183.143408:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143415:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076694272:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767183.143415:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143425:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767183.143431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767183.143436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767183.143446:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076694656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767183.143455:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767183.143465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767183.143468:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7d880 x1862023076694400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767183.143486:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767183.143493:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00100000:0.0:1775767183.143495:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076694400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767183.143502:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.143506:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767183.143508:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767183.143510:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694400:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767183.143517:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143518:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694528:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767183.143525:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775767183.143542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767183.143546:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076694784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767183.143553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7ce00 x1862023076694528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767183.143570:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076694528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143590:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767183.143593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767183.143594:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767183.143598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767183.143603:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:2.0:1775767183.143611:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143617:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694656:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775767183.143622:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767183.143638:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076694912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767183.143645:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767183.143653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767183.143658:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767183.143661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7ed80 x1862023076694656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143665:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00080000:2.0:1775767183.143671:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143675:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076694656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767183.143679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767183.143696:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076695040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767183.143701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.143709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767183.143719:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694784:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767183.143736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775767183.143743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767183.143744:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767183.143751:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3684 00000100:00100000:0.0:1775767183.143753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7f800 x1862023076694784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143759:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767183.143765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076694784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767183.143767:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767183.143778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767183.143778:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767183.143784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076694784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767183.143797:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076695168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767183.143803:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694912:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767183.143811:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767183.143815:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767183.143816:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767183.143821:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3684 00000800:00000100:0.0:1775767183.143824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767183.143829:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767183.143834:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7ea00 x1862023076694912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767183.143840:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767183.143855:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076695296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767183.143858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076694912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767183.143866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.143874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076694912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767183.143890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767183.143905:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7df80 x1862023076695040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767183.143919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076695040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767183.143924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.143926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767183.143936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767183.143946:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143951:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7fb80 x1862023076695168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767183.143958:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076695168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767183.143963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.143964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.143966:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767183.143969:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767183.143978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767183.143990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767183/real 1775767183] req@ffffa0434aa7f800 x1862023076695296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767183.143997:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076695296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767183.144002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767183.144004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767183.144006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076695296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767183.335110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767183.335110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767184.167122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767184.167151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767184.359147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767184.359147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767185.191119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767185.191154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767185.383111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767185.383134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767186.215124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767186.215287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767186.407126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767186.407126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767187.239103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767187.239128:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767187.431132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767187.431132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767188.263151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767188.263153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767188.263164:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263173:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263198:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775767188.263203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767188.263204:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767188.263248:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076695424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767188.263269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767188.263273:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263276:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263278:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263281:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263289:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767188.263300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076695552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767188.263304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076695424:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767188.263312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767188.263316:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263318:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263320:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263322:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263325:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767188.263334:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076695680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767188.263340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263349:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767188.263350:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263353:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263356:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263359:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263364:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767188.263374:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767188.263378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076695808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767188.263389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767188.263394:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263399:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263404:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263408:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263414:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767188.263420:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7ed80 x1862023076695424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767188.263430:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076695936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767188.263438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076695424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767188.263444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767188.263454:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767188.263457:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767188.263459:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.263466:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076695424:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767188.263467:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263486:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263491:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767188.263501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695552:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767188.263505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767188.263537:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076696064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767188.263540:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767188.263561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767188.263564:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7ce00 x1862023076695552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767188.263568:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767188.263572:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263583:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263589:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263598:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263606:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767188.263609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076695552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.263617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767188.263625:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076696192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767188.263627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767188.263638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767188.263650:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767188.263654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767188.263657:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767188.263663:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3689 00000800:00000100:0.0:1775767188.263664:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767188.263671:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767188.263676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7d880 x1862023076695680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767188.263678:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263687:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767188.263695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076695680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767188.263703:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076696320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767188.263709:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263719:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767188.263721:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767188.263726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767188.263726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767188.263733:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767188.263737:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695808:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767188.263739:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3689 00000100:00080000:2.0:1775767188.263746:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767188.263750:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767188.263755:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767188.263765:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767188.263768:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076696448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767188.263778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7d500 x1862023076695808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.263789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076695808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.263805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.263809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767188.263814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767188.263827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767188.263834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7e300 x1862023076695936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.263841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076695936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.263850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.263856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076695936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767188.263875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767188.263896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767188.263907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7f480 x1862023076696064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.263916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076696064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.263921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.263931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767188.263935:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767188.263949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767188.263956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7e680 x1862023076696192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.263972:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076696192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.263977:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.263979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.263981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767188.263985:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767188.263995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767188.264001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7c380 x1862023076696320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.264008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076696320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.264012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.264014:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.264016:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767188.264019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767188.264059:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767188.264073:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767188/real 1775767188] req@ffffa0434aa7dc00 x1862023076696448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767188.264088:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076696448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767188.264094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767188.264096:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767188.264098:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076696448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767188.455139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767188.455139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767189.287116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767189.287137:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767189.479115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767189.479115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767190.311143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767190.311286:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767190.503113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767190.503114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767191.335114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767191.335139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767191.527134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767191.527134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767192.359134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767192.359179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767192.551113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767192.551113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767193.383136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767193.383142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767193.383155:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767193.383161:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383172:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3695 00010000:00010000:3.0:1775767193.383180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767193.383183:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767193.383191:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767193.383227:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076696576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767193.383252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767193.383266:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383271:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:2.0:1775767193.383276:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767193.383281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076696576:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767193.383284:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767193.383315:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076696704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767193.383348:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767193.383357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767193.383364:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383378:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:2.0:1775767193.383384:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767193.383388:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383395:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767193.383401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7f100 x1862023076696576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767193.383413:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076696832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383423:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076696576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767193.383424:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383433:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767193.383438:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383447:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:0.0:1775767193.383448:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.383456:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767193.383456:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767193.383463:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076696576:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767193.383478:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383486:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767193.383499:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076696960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383502:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767193.383516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383518:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767193.383520:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383524:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3695 00000800:00000100:0.0:1775767193.383533:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767193.383537:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767193.383543:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767193.383547:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7c000 x1862023076696704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767193.383548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767193.383562:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076697088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076696704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767193.383596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767193.383606:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383612:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:2.0:1775767193.383617:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767193.383621:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767193.383622:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767193.383624:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775767193.383629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767193.383636:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076697216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383637:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767193.383651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767193.383654:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767193.383656:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767193.383660:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383668:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:2.0:1775767193.383671:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767193.383673:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383677:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767193.383682:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767193.383688:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076697344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383695:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7c700 x1862023076696832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.383717:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076696832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767193.383718:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383726:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767193.383729:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767193.383736:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767193.383737:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:0.0:1775767193.383743:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767193.383744:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767193.383749:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767193.383751:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383760:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767193.383770:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076697472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767193.383772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696960:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767193.383787:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767193.383790:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767193.383792:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767193.383797:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3695 00000100:00080000:2.0:1775767193.383802:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767193.383807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767193.383808:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767193.383815:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767193.383819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7dc00 x1862023076696960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.383828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076696960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767193.383834:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076697600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767193.383863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.383868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.383871:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076696960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767193.383877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767193.383898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767193.383906:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7c380 x1862023076697088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.383915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076697088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767193.383924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.383927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.383930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767193.383936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767193.383957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767193.383967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7e680 x1862023076697216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.383980:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076697216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767193.383986:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.383988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.383995:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767193.384000:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767193.384015:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767193.384022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7f480 x1862023076697344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.384050:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076697344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767193.384055:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.384057:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.384059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767193.384072:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767193.384085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767193.384091:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7e300 x1862023076697472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.384099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076697472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767193.384104:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.384105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.384107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767193.384110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767193.384119:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767193.384133:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767193/real 1775767193] req@ffffa0434aa7d500 x1862023076697600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767193.384141:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076697600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767193.384146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767193.384148:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767193.384150:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076697600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767193.575106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767193.575106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767194.407139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767194.407167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767194.599108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767194.599113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767195.431119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767195.431169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767195.623097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767195.623103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767196.455149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767196.455185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767196.647129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767196.647135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767197.479137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767197.479174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767197.671086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767197.671096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767198.503152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767198.503215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767198.503215:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503222:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767198.503227:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503236:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503247:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503254:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767198.503302:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076697728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767198.503325:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503328:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767198.503329:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503333:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503337:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503340:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503349:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767198.503356:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076697728:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767198.503363:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076697856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767198.503390:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503392:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767198.503394:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503397:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503401:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503404:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503408:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767198.503417:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076697984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767198.503446:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503450:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767198.503452:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503457:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503462:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503466:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767198.503472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503473:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767198.503489:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076698112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767198.503499:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503502:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767198.503505:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503509:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00100000:0.0:1775767198.503521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825ea00 x1862023076697728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767198.503525:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503535:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503540:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767198.503544:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076697728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.503554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767198.503555:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076698240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767198.503563:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767198.503570:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767198.503571:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076697728:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767198.503584:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767198.503586:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503590:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503593:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503597:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767198.503601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697856:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767198.503602:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767198.503615:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076698368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767198.503623:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767198.503627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767198.503629:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767198.503632:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503636:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503640:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767198.503641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825e680 x1862023076697856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767198.503644:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503651:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767198.503663:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076698496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767198.503683:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076697856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.503713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767198.503713:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503719:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767198.503720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767198.503722:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767198.503725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767198.503729:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503735:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767198.503737:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767198.503741:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767198.503747:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767198.503758:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076698624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767198.503764:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767198.503771:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767198.503775:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775767198.503778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825c000 x1862023076697984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767198.503778:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767198.503786:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3700 00000100:00080000:3.0:1775767198.503791:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767198.503796:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767198.503799:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076697984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767198.503801:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767198.503810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767198.503814:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076698752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767198.503816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.503822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076697984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767198.503834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076698112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767198.503850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.503856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825ed80 x1862023076698112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.503864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076698112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.503877:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.503880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.503882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076698112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767198.503886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076698240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767198.503899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.503906:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825e300 x1862023076698240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.503913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076698240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.503919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.503920:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.503922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076698240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767198.503936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767198.503947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.503953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825f100 x1862023076698368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.503960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076698368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.503966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.503968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.503976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767198.503981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767198.504002:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.504013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825d500 x1862023076698496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.504048:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076698496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.504055:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.504058:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.504061:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767198.504065:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767198.504081:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.504089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825dc00 x1862023076698624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.504096:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076698624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.504102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.504104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.504106:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767198.504109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767198.504118:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767198.504131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767198/real 1775767198] req@ffffa0435825d880 x1862023076698752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767198.504139:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076698752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767198.504145:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767198.504147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767198.504149:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076698752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767198.695150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767198.695153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767199.527131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767199.527165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767199.719158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767199.719161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767200.551156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767200.551163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767200.743091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767200.743103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767201.575104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767201.575124:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767201.767135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767201.767135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767202.599169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767202.599216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767202.791133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767202.791214:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767203.623139:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623148:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767203.623153:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767203.623154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767203.623154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767203.623163:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623187:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623193:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767203.623224:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076698880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623247:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623250:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767203.623251:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767203.623254:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623256:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623259:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623267:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767203.623275:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076699008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623283:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623286:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767203.623288:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767203.623292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076698880:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767203.623292:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623297:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623302:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623305:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767203.623321:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076699136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623331:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623340:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767203.623342:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767203.623347:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623352:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623358:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623364:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767203.623369:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767203.623406:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076699264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623414:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623416:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767203.623417:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767203.623420:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623424:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623427:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767203.623431:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350932300 x1862023076698880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767203.623433:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767203.623445:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076699392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623452:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623455:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767203.623456:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767203.623459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076698880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767203.623462:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623466:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623472:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767203.623479:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767203.623480:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775767203.623487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767203.623492:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076699520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767203.623496:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076698880:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767203.623509:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623511:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767203.623513:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767203.623516:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623519:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623521:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623524:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767203.623532:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767203.623533:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076699648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623547:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623550:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767203.623551:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767203.623554:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623556:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623559:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623562:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767203.623569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767203.623571:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076699776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767203.623579:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767203.623582:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767203.623587:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767203.623588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350930e00 x1862023076699008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767203.623595:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3705 00000100:00080000:1.0:1775767203.623601:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767203.623606:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767203.623610:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767203.623612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076699008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767203.623619:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076699904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767203.623642:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.623648:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.623653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767203.623660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767203.623682:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.623694:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350930000 x1862023076699136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.623717:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076699136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.623724:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.623727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.623730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767203.623735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767203.623754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.623765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350931880 x1862023076699264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.623776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076699264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.623783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.623786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.623806:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767203.623813:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767203.623835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.623847:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350932a00 x1862023076699392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.623859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076699392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.623868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.623871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.623875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076699392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767203.623880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767203.623899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.623908:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350930a80 x1862023076699520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.623920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076699520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.623928:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.623931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.623934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767203.623970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767203.623989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.624008:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350930700 x1862023076699648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.624021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076699648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.624029:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.624031:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.624034:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767203.624040:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767203.624082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.624093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350932680 x1862023076699776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.624121:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076699776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.624129:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.624133:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.624136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767203.624142:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767203.624159:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767203.624168:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767203/real 1775767203] req@ffffa04350931c00 x1862023076699904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767203.624179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076699904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767203.624192:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767203.624195:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767203.624198:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076699904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767203.815137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767203.815138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767204.647116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767204.647180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767204.839081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767204.839086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767205.671153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767205.671188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767205.863186:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767205.863186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767206.695126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767206.695147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767206.887113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767206.887131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767207.719152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767207.719201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767207.911168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767207.911182:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767208.743106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767208.743145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767208.743155:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743165:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3710 00010000:00010000:3.0:1775767208.743173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767208.743182:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743193:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767208.743227:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076700032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767208.743251:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767208.743256:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743259:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743262:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767208.743282:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076700160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767208.743290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767208.743310:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767208.743311:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076700032:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767208.743316:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743339:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743343:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767208.743378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076700288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767208.743388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767208.743394:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743397:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743410:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743413:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743418:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767208.743425:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767208.743447:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076700416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767208.743456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767208.743460:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743463:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743466:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743469:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743473:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767208.743495:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076700544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767208.743495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349641500 x1862023076700032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767208.743534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767208.743542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767208.743543:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076700032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767208.743553:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743562:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767208.743565:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767208.743569:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767208.743574:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775767208.743575:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767208.743583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076700032:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767208.743590:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076700672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767208.743598:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767208.743603:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767208.743608:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743613:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743615:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743618:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743622:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767208.743633:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076700800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767208.743637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767208.743649:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743653:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775767208.743658:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349642300 x1862023076700160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767208.743668:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767208.743677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076700160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767208.743684:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743694:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743710:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743720:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767208.743725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767208.743734:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076700928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767208.743736:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.743747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767208.743750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767208.743757:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767208.743758:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767208.743762:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3710 00000100:00080000:0.0:1775767208.743766:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767208.743769:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767208.743772:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767208.743779:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076701056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767208.743790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767208.743822:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.743837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349641880 x1862023076700288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.743853:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076700288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.743863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.743867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.743871:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767208.743879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767208.743904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.743917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349643800 x1862023076700416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.743932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076700416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.743942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.743946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.743957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767208.743965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767208.743991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.744004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349643b80 x1862023076700544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.744019:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076700544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.744030:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.744033:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.744038:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076700544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767208.744045:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767208.744097:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.744111:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349640a80 x1862023076700672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.744126:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076700672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.744136:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.744141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.744144:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767208.744152:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767208.744175:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.744201:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349643100 x1862023076700800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.744216:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076700800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.744226:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.744230:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.744234:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767208.744241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767208.744266:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.744279:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349640380 x1862023076700928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.744295:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076700928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.744305:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.744309:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.744312:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076700928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767208.744360:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767208.744386:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767208.744400:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767208/real 1775767208] req@ffffa04349642a00 x1862023076701056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767208.744416:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076701056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767208.744433:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767208.744438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767208.744442:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767208.935102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767208.935110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767209.767160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767209.767207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767209.959092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767209.959093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767210.791117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767210.791167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767210.983102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767210.983102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767211.815173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767211.815177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767212.007122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767212.007122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767212.839181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767212.839208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767213.031107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767213.031109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767213.863140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767213.863161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767213.863255:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767213.863263:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863272:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863291:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863298:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767213.863334:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076701184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863360:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767213.863362:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863364:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863367:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863370:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863376:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767213.863384:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076701312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767213.863409:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863413:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863416:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863420:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076701184:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767213.863425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767213.863436:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076701440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767213.863447:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863450:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863452:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863454:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767213.863467:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076701568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863476:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767213.863481:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863485:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863489:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863500:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1775767213.863506:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767213.863522:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076701696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767213.863533:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863536:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863539:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863542:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863547:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767213.863552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350931c00 x1862023076701184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767213.863556:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076701824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767213.863588:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863591:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00080000:1.0:1775767213.863594:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863596:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863599:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767213.863601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076701184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767213.863606:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076701952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767213.863634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767213.863636:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767213.863642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767213.863649:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3715 00000100:00100000:2.0:1775767213.863651:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076701184:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767213.863655:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863662:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863665:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701312:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767213.863670:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767213.863684:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076702080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767213.863695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767213.863698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767213.863700:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767213.863705:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3715 00000800:00000100:2.0:1775767213.863710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767213.863711:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767213.863717:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767213.863721:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767213.863730:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076702208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767213.863741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350932680 x1862023076701312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.863759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076701312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.863770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.863774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.863778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767213.863807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767213.863831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863842:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350930700 x1862023076701440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.863851:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076701440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.863857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.863859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.863861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767213.863865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767213.863878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350930a80 x1862023076701568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.863903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076701568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.863909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.863911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.863913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767213.863917:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767213.863929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350932a00 x1862023076701696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.863942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076701696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.863947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.863949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.863951:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076701696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767213.863957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767213.863975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.863986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350931880 x1862023076701824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.863999:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076701824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.864020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.864023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.864048:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767213.864054:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767213.864078:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.864090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350930000 x1862023076701952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.864103:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076701952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.864112:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.864115:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.864118:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076701952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767213.864123:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767213.864138:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.864145:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350930e00 x1862023076702080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.864152:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076702080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.864157:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.864159:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.864167:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767213.864171:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767213.864185:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767213.864191:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767213/real 1775767213] req@ffffa04350932300 x1862023076702208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767213.864198:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076702208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767213.864203:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767213.864205:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767213.864207:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767214.055130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767214.055130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767214.887149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767214.887166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767215.079094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767215.079116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767215.911091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767215.911140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767216.103155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767216.103155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767216.935113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767216.935132:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767217.127108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767217.127109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767217.959108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767217.959140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767218.151108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767218.151116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767218.983099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767218.983165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767218.983205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767218.983217:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983245:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983255:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983262:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767218.983310:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076702336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767218.983337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983342:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767218.983344:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767218.983350:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076702336:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767218.983351:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983360:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983365:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767218.983391:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076702464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767218.983404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767218.983414:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767218.983415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767218.983424:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983430:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983434:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983440:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767218.983454:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076702592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767218.983460:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349640e00 x1862023076702336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767218.983470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767218.983480:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983493:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983502:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983511:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983517:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767218.983523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076702336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767218.983530:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076702720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767218.983541:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767218.983543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767218.983551:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983555:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983558:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767218.983558:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767218.983564:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983567:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767218.983569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076702336:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767218.983577:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076702848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767218.983584:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767218.983588:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983590:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983593:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767218.983596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767218.983598:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983605:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767218.983616:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076702976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767218.983629:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767218.983633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767218.983638:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983642:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983646:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767218.983646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349640700 x1862023076702464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767218.983654:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983663:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767218.983666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076702464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.983674:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767218.983676:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076703104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767218.983680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.983687:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767218.983690:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983694:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767218.983696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702592:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767218.983697:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983706:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:0.0:1775767218.983711:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983716:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767218.983722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767218.983723:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767218.983736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349642680 x1862023076702592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767218.983740:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076703232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767218.983756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076702592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767218.983760:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767218.983766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767218.983768:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767218.983777:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3720 00000100:00080000:2.0:1775767218.983782:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.983786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.983789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702592:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767218.983790:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767218.983799:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767218.983799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702720:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767218.983807:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767218.983820:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076703360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767218.983826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.983838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349641180 x1862023076702720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.983847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076702720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.983852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.983856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.983860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767218.983866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767218.983884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.983904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349641c00 x1862023076702848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.983918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076702848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.983925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.983927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.983935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076702848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767218.983950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767218.983966:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.983973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349640e00 x1862023076702976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.983980:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076702976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.983985:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.983987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.983990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076702976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767218.983993:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767218.984002:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.984007:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349642a00 x1862023076703104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.984014:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076703104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.984019:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.984021:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.984022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767218.984042:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767218.984056:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.984070:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349640380 x1862023076703232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.984078:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076703232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.984084:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.984086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.984090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767218.984094:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767218.984111:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767218.984123:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767218/real 1775767218] req@ffffa04349643100 x1862023076703360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767218.984135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076703360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767218.984143:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767218.984147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767218.984150:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076703360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767219.175119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767219.175119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767220.007085:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767220.007156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767220.199106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767220.199108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767221.031149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767221.031150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767221.223090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767221.223102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767222.055126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767222.055212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767222.247098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767222.247102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767223.079116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767223.079184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767223.271172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767223.271255:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767224.103155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767224.103222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103231:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1775767224.103234:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767224.103236:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103250:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103263:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103270:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767224.103317:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076703488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767224.103370:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103374:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103377:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103381:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103391:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767224.103402:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076703616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767224.103412:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076703488:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767224.103421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767224.103424:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103428:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103431:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103433:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103438:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767224.103450:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076703744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767224.103462:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103465:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103475:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103478:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103483:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767224.103492:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076703872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767224.103506:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767224.103517:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103521:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767224.103522:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103526:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103530:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103534:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103537:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767224.103547:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076704000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103553:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775767224.103558:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350932300 x1862023076703488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767224.103559:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103568:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103580:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103589:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103594:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767224.103602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076703488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767224.103606:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076704128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767224.103657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.103673:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767224.103680:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767224.103684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076703488:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767224.103685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767224.103692:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103698:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103702:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103707:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103712:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767224.103718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767224.103724:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076704256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767224.103738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767224.103741:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103744:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103748:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103756:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767224.103756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767224.103765:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767224.103776:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076704384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767224.103786:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767224.103786:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350930e00 x1862023076703616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767224.103797:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767224.103799:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767224.103806:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3725 00000100:00080000:1.0:1775767224.103815:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767224.103827:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767224.103828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076703616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767224.103839:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767224.103842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.103847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767224.103853:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076704512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767224.103859:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767224.103882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767224.103909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.103933:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350930000 x1862023076703744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.103941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076703744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.103947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.103949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.103952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767224.103956:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767224.103978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.103985:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350931880 x1862023076703872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.103992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076703872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.103998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076703872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767224.104006:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767224.104016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.104021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350932a00 x1862023076704000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.104048:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076704000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.104060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104065:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767224.104069:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767224.104085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.104092:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350930a80 x1862023076704128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.104100:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076704128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.104105:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104109:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767224.104123:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767224.104134:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.104140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350930700 x1862023076704256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.104147:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076704256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.104152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104160:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767224.104165:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767224.104206:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.104219:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350932680 x1862023076704384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.104233:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076704384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.104241:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104244:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104246:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767224.104251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767224.104266:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767224.104273:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767224/real 1775767224] req@ffffa04350931c00 x1862023076704512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767224.104281:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076704512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767224.104286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767224.104289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767224.104291:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076704512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767224.295112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767224.295127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767225.127168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767225.127224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767225.319131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767225.319131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767226.151109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767226.151289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767226.343152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767226.343153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767227.175178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767227.175246:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767227.367189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767227.367189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767228.199115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767228.199231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767228.391130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767228.391130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767229.223126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767229.223127:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767229.223153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767229.223158:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767229.223170:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:1.0:1775767229.223186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767229.223195:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767229.223245:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076704640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767229.223282:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767229.223288:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:3.0:1775767229.223289:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767229.223291:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767229.223299:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:1.0:1775767229.223305:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767229.223312:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223314:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076704640:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767229.223328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767229.223348:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076704768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767229.223362:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767229.223387:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767229.223390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767229.223394:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767229.223399:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:1.0:1775767229.223407:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767229.223412:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767229.223419:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767229.223441:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350931c00 x1862023076704640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767229.223445:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076704896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767229.223458:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076704640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767229.223474:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767229.223478:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767229.223481:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767229.223483:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767229.223487:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076704640:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767229.223492:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:1.0:1775767229.223498:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767229.223503:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767229.223510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767229.223514:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767229.223530:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076705024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767229.223537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767229.223550:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767229.223551:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350932680 x1862023076704768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767229.223566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767229.223571:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076704768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223576:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:1.0:1775767229.223585:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767229.223587:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767229.223593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767229.223595:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704768:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767229.223613:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767229.223614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767229.223636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076705152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767229.223640:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350930700 x1862023076704896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223671:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767229.223674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076704896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223680:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767229.223685:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767229.223688:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767229.223693:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:2.0:1775767229.223694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767229.223699:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076704896:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767229.223702:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767229.223709:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705024:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767229.223716:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767229.223733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767229.223736:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076705280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767229.223744:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350930a80 x1862023076705024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223759:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767229.223764:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076705024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223765:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767229.223769:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767229.223779:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:2.0:1775767229.223779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767229.223787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767229.223788:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767229.223791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705024:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767229.223797:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705152:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767229.223806:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767229.223816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767229.223824:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076705408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767229.223835:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350932a00 x1862023076705152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223837:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767229.223846:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767229.223851:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076705152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223857:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767229.223867:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767229.223868:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:2.0:1775767229.223873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767229.223875:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767229.223876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705152:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767229.223884:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223891:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705280:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767229.223892:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767229.223911:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076705536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767229.223916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767229.223927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350931880 x1862023076705280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223930:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767229.223940:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767229.223942:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767229.223943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076705280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767229.223953:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3730 00000100:00080000:2.0:1775767229.223956:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767229.223958:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767229.223963:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767229.223966:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767229.223971:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767229.223980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767229.223986:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076705664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767229.223992:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767229.224024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767229.224061:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350930000 x1862023076705408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767229.224074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076705408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767229.224088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767229.224090:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767229.224093:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767229.224103:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767229.224119:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767229.224126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350930e00 x1862023076705536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767229.224152:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076705536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767229.224158:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767229.224160:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767229.224162:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767229.224167:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767229.224181:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767229.224188:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767229/real 1775767229] req@ffffa04350932300 x1862023076705664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767229.224195:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076705664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767229.224201:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767229.224202:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767229.224210:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076705664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767229.415152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767229.415157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767230.247107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767230.247208:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767230.439131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767230.439135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767231.271164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767231.271190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767231.463158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767231.463158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767232.295137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767232.295193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767232.487114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767232.487165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767233.319142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767233.319174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767233.511103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767233.511103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767234.343152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767234.343257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767234.343266:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343275:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3736 00010000:00010000:1.0:1775767234.343276:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767234.343284:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343290:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767234.343324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076705792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767234.343351:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343354:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343357:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343359:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767234.343378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076705920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767234.343385:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076705792:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767234.343406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767234.343410:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343415:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343419:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343422:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767234.343436:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076706048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767234.343446:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343449:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343451:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343454:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767234.343464:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076706176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767234.343475:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343477:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343485:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767234.343487:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767234.343491:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767234.343509:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076706304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767234.343519:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767234.343526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767234.343529:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343533:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343537:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343540:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343544:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767234.343547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767234.343557:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076706432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343567:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767234.343568:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767234.343570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706304:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767234.343574:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343579:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767234.343581:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343585:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767234.343593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767234.343596:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076706560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343612:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767234.343617:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343621:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343625:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767234.343627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767234.343631:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343636:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767234.343650:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076706688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767234.343663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767234.343667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767234.343669:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767234.343674:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3736 00000100:00080000:0.0:1775767234.343679:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767234.343682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706688:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767234.343686:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767234.343693:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767234.343708:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076706816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767234.343733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767234.535112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767234.535112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767235.367131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767235.367238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767235.559145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767235.559145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767236.391156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767236.391318:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767236.583160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767236.583161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767237.415184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767237.415354:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775767237.415430:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775767237.415475:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767237.415494:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415554:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415568:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415604:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415613:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415620:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415628:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415636:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767237.415643:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767237.415852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349642a00 x1862023076706816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.415879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076706816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.415893:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.415899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.415904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767237.415913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349640380 x1862023076706688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.415937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076706688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.415947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.415950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.415954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767237.415962:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349643100 x1862023076706560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.415975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076706560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.415996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416001:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767237.416012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349642300 x1862023076706304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416069:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076706304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416084:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416088:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767237.416096:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349641500 x1862023076706432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076706432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416124:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416129:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076706432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767237.416140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349643b80 x1862023076705920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416150:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076705920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416167:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076705920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767237.416179:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349643800 x1862023076706048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416192:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076706048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416201:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416204:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416207:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767237.416214:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349641880 x1862023076706176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416225:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076706176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416234:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416240:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076706176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767237.416248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767234/real 1775767237] req@ffffa04349640a80 x1862023076705792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767237.416261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076705792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767237.416276:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767237.416280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767237.416284:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076705792:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767237.607119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767237.607120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767238.439140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767238.439257:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767238.631104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767238.631117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767239.463144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767239.463172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767239.463182:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463192:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463200:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463206:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767239.463241:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076706944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775767239.463248:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767239.463271:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767239.463278:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463284:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463289:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463296:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767239.463296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076706944:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767239.463310:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767239.463329:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076707072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767239.463342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1775767239.463371:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767239.463374:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463381:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463388:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463393:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767239.463401:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767239.463418:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350932300 x1862023076706944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767239.463419:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076707200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767239.463451:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463455:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767239.463457:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463464:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00100000:2.0:1775767239.463466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076706944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767239.463470:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463479:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767239.463487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767239.463488:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767239.463496:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.463502:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076706944:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767239.463506:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076707328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767239.463519:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463524:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767239.463525:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463530:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00100000:2.0:1775767239.463535:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767239.463537:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463546:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767239.463561:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1775767239.463569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767239.463579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076707456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767239.463588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350930e00 x1862023076707072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767239.463594:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767239.463607:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767239.463612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076707072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767239.463613:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463618:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463625:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767239.463635:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767239.463640:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767239.463650:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076707584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767239.463660:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463664:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767239.463666:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767239.463668:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767239.463671:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463677:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767239.463679:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767239.463685:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767239.463693:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767239.463693:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767239.463712:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076707712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767239.463723:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767239.463728:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1775767239.463729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767239.463739:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767239.463745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350930000 x1862023076707200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767239.463747:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463756:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463763:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767239.463769:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076707200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767239.463772:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767239.463786:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767239.463792:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076707840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767239.463793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.463803:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767239.463815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767239.463817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707328:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767239.463823:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767239.463827:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767239.463833:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3741 00000100:00080000:1.0:1775767239.463838:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767239.463843:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767239.463850:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767239.463857:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767239.463864:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076707968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767239.463874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350931880 x1862023076707328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.463889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076707328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.463898:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.463903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.463922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767239.463941:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767239.463965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767239.463977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350932a00 x1862023076707456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.463989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076707456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.463998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.464001:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.464004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076707456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767239.464009:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767239.464076:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767239.464088:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350930a80 x1862023076707584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.464101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076707584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.464110:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.464114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.464117:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767239.464124:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767239.464147:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767239.464167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350930700 x1862023076707712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.464179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076707712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.464187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.464190:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.464193:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767239.464198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767239.464217:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767239.464228:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350932680 x1862023076707840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.464238:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076707840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.464245:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.464248:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.464251:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767239.464256:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767239.464273:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767239.464282:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767239/real 1775767239] req@ffffa04350931c00 x1862023076707968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767239.464293:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076707968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767239.464306:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767239.464310:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767239.464313:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076707968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767239.655114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767239.655114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767240.487129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767240.487258:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767240.679156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767240.679251:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767241.511122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767241.511190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767241.703120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767241.703122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767242.535106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767242.535160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767242.727096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767242.727096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767243.559128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767243.559166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767243.751099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767243.751099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767244.583111:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767244.583115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767244.583122:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767244.583126:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583137:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583148:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767244.583154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767244.583156:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767244.583196:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076708096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767244.583235:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767244.583241:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583250:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583255:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583265:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767244.583268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076708096:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767244.583279:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076708224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767244.583289:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583292:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767244.583294:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583297:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583301:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583304:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583309:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767244.583318:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076708352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767244.583324:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767244.583328:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583331:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767244.583332:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583335:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583344:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583348:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583354:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767244.583358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350931c00 x1862023076708096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767244.583364:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076708480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767244.583373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767244.583375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076708096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767244.583376:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767244.583378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583382:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583386:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767244.583386:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767244.583389:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767244.583390:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767244.583393:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767244.583394:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076708096:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767244.583401:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076708608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767244.583409:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583413:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775767244.583414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767244.583415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583418:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583421:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583425:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583430:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767244.583438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076708736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767244.583442:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583455:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350932680 x1862023076708224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767244.583460:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767244.583465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767244.583466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076708224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767244.583469:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583473:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583476:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767244.583476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767244.583481:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767244.583482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767244.583488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767244.583491:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076708864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767244.583500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767244.583503:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767244.583505:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767244.583505:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767244.583508:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00080000:1.0:1775767244.583512:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767244.583514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350930700 x1862023076708352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767244.583515:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583520:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767244.583526:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076708352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767244.583528:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076708992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767244.583536:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767244.583538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767244.583539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767244.583542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767244.583542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708352:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767244.583544:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767244.583548:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3746 00000100:00100000:2.0:1775767244.583548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708480:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767244.583556:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767244.583560:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767244.583564:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767244.583565:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767244.583573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350930a80 x1862023076708480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767244.583575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076709120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767244.583600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076708480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583609:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583612:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767244.583621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767244.583644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350932a00 x1862023076708608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767244.583667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076708608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076708608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767244.583701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767244.583719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583727:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350931880 x1862023076708736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767244.583734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076708736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583753:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767244.583756:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767244.583769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583776:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350930000 x1862023076708864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767244.583783:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076708864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767244.583794:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767244.583803:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350930e00 x1862023076708992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767244.583815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076708992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076708992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767244.583832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076709120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767244.583844:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767244.583849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767244/real 1775767244] req@ffffa04350932300 x1862023076709120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767244.583856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076709120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767244.583861:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767244.583863:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767244.583865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076709120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767244.775134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767244.775134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767245.607117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767245.607175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767245.799108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767245.799112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767246.631118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767246.631150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767246.823126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767246.823126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775767247.655136:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767247.655166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767247.847130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767247.847130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767248.679163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767248.679221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767248.871151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767248.871161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767249.703112:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703121:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767249.703125:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775767249.703127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767249.703131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767249.703134:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703144:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703150:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767249.703187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076709248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767249.703214:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703219:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767249.703221:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703225:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703229:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703234:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703244:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767249.703258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076709248:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767249.703262:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076709376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767249.703293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703297:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767249.703299:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703304:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703309:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703312:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703317:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767249.703327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767249.703328:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076709504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767249.703337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703340:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767249.703342:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703345:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703351:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703353:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767249.703363:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076709632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767249.703370:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703375:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767249.703376:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767249.703376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350932300 x1862023076709248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767249.703382:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703392:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703396:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703401:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767249.703406:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076709248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767249.703411:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076709760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767249.703427:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703429:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767249.703431:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703451:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703455:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703460:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703466:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767249.703479:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767249.703480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076709888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767249.703492:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767249.703494:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703514:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767249.703518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767249.703518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076709248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767249.703524:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703529:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703532:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767249.703541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076710016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767249.703553:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767249.703562:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703564:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767249.703565:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703569:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703572:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703579:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767249.703590:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076710144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767249.703595:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767249.703602:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767249.703605:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767249.703606:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767249.703609:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3751 00000100:00080000:1.0:1775767249.703611:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767249.703614:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767249.703618:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767249.703618:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350930e00 x1862023076709376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767249.703627:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076710272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767249.703652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076709376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.703663:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.703667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.703670:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767249.703677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767249.703700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.703711:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350930000 x1862023076709504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.703725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076709504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.703734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.703737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.703741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767249.703755:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767249.703777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.703788:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350931880 x1862023076709632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.703801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076709632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.703809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.703813:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.703817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767249.703823:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767249.703844:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.703855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350932a00 x1862023076709760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.703867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076709760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.703875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.703879:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.703882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076709760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767249.703889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076709888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767249.703907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.703917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350930a80 x1862023076709888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.703938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076709888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.703968:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.703972:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.703976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076709888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767249.703996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767249.704019:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.704060:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350930700 x1862023076710016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.704074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076710016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.704083:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.704086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.704090:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767249.704096:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767249.704115:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.704126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350932680 x1862023076710144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.704138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076710144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.704180:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.704185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.704189:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767249.704197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767249.704224:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767249.704237:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767249/real 1775767249] req@ffffa04350931c00 x1862023076710272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767249.704251:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076710272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767249.704262:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767249.704266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767249.704270:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076710272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767249.895109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767249.895110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767250.727161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767250.727218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767250.919126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767250.919126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767251.751124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767251.751157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767251.943077:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767251.943090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767252.775138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767252.775274:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767252.967106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767252.967107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767253.799130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767253.799200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767253.991103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767253.991103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767254.823134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767254.823136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823143:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767254.823147:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823156:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823178:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823185:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767254.823192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767254.823219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076710400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823241:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767254.823248:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823254:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00100000:2.0:1775767254.823258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076710400:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767254.823261:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823269:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767254.823296:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076710528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823308:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767254.823313:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823316:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823322:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823326:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767254.823331:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767254.823352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076710656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823361:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767254.823370:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350931c00 x1862023076710400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767254.823374:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823383:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823390:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823393:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076710400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767254.823398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767254.823410:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767254.823412:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076710784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767254.823416:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823423:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076710400:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767254.823431:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767254.823436:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823439:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823441:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823443:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823447:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767254.823450:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767254.823458:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076710912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823467:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823469:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767254.823470:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823473:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823476:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767254.823480:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767254.823483:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823492:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767254.823500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350932680 x1862023076710528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767254.823503:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076711040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823512:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823521:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767254.823525:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823532:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00100000:2.0:1775767254.823535:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076710528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767254.823538:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767254.823547:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767254.823548:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767254.823554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823557:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710528:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767254.823558:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767254.823565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767254.823576:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076711168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823584:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767254.823588:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823592:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3756 00000800:00000100:2.0:1775767254.823597:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767254.823598:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767254.823604:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823608:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767254.823616:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350930700 x1862023076710656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767254.823619:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076711296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767254.823650:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767254.823653:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767254.823662:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767254.823666:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3756 00000100:00080000:1.0:1775767254.823670:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767254.823673:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076710656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767254.823676:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767254.823683:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767254.823694:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076711424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767254.823698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.823709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767254.823727:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767254.823748:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350930a80 x1862023076710784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.823764:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076710784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.823770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.823772:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767254.823785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767254.823796:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350932a00 x1862023076710912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.823809:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076710912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.823823:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.823825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076710912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767254.823834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767254.823855:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350931880 x1862023076711040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.823880:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076711040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.823889:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.823906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823910:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767254.823915:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767254.823930:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350930000 x1862023076711168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.823952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076711168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.823957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.823959:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.823966:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767254.823970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767254.823988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.823997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350930e00 x1862023076711296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.824009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076711296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.824017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.824020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.824024:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767254.824055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767254.824078:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767254.824086:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767254/real 1775767254] req@ffffa04350932300 x1862023076711424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767254.824094:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076711424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767254.824099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767254.824101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767254.824103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076711424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767255.015118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767255.015119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767255.847158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767255.847233:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767256.039105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767256.039133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767256.871159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767256.871212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767257.063156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767257.063163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767257.895110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767257.895239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767258.087138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767258.087154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767258.919162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767258.919206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767259.111115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767259.111130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767259.943172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767259.943173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767259.943186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767259.943195:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943203:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943211:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943218:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767259.943259:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076711552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943290:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943293:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767259.943294:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943298:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943302:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943306:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943318:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076711552:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767259.943322:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767259.943336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076711680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767259.943349:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943352:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943369:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767259.943374:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767259.943375:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943382:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767259.943393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076711808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767259.943403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943405:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943408:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943410:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943414:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767259.943423:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7d880 x1862023076711552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767259.943424:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076711936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943458:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767259.943459:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943461:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943463:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943466:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767259.943473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076711552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767259.943478:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076712064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767259.943492:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943499:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943506:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076711552:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767259.943511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943514:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767259.943516:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943519:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943522:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943525:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943529:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767259.943533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767259.943540:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076712192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767259.943552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943555:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943557:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943559:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767259.943566:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767259.943575:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076712320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767259.943582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767259.943586:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7ce00 x1862023076711680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767259.943587:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943594:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943603:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767259.943604:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076711680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767259.943610:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767259.943619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767259.943619:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767259.943626:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767259.943632:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076712448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767259.943641:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767259.943644:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767259.943647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775767259.943649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711808:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767259.943649:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767259.943655:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3761 00000100:00080000:2.0:1775767259.943657:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767259.943659:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767259.943663:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767259.943671:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076712576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767259.943676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7ed80 x1862023076711808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076711808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943706:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943708:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943710:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767259.943714:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767259.943725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7f800 x1862023076711936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943739:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076711936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943743:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943745:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076711936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767259.943750:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767259.943760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7fb80 x1862023076712064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076712064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943785:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767259.943802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767259.943814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7df80 x1862023076712192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076712192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943836:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767259.943839:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767259.943870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7ea00 x1862023076712320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943885:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076712320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943897:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767259.943903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767259.943916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7d880 x1862023076712448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076712448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767259.943942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767259.943951:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767259.943956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767259/real 1775767259] req@ffffa0434aa7d500 x1862023076712576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767259.943962:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076712576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767259.943967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767259.943969:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767259.943975:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076712576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767260.135201:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767260.135201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767260.967145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767260.967189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767261.159146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767261.159147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767261.991155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767261.991452:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767262.183116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767262.183116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767263.015161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767263.015267:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767263.207099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767263.207104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767264.039156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767264.039231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767264.231136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767264.231138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767265.063149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767265.063164:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767265.063172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767265.063175:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063196:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063206:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767265.063270:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767265.063273:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076712704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767265.063300:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063304:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063309:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063313:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767265.063322:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767265.063332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076712704:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767265.063333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076712832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767265.063358:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063363:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063367:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063372:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767265.063377:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767265.063390:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076712960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767265.063418:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063430:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3766 00000800:00000100:0.0:1775767265.063438:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767265.063440:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063447:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767265.063452:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767265.063464:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076713088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063472:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063476:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767265.063478:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063484:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063489:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767265.063500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350932300 x1862023076712704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767265.063505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767265.063532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076713216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767265.063537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076712704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767265.063558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063575:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767265.063577:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767265.063579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767265.063584:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063590:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767265.063593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767265.063598:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767265.063602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076712704:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767265.063606:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767265.063621:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076713344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767265.063644:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767265.063648:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063662:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063667:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767265.063671:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767265.063676:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767265.063688:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076713472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767265.063694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767265.063707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767265.063711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767265.063712:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063715:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063719:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767265.063723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350930e00 x1862023076712832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767265.063725:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767265.063732:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767265.063758:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076712832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767265.063758:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076713600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767265.063794:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767265.063818:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767265.063819:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767265.063821:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767265.063825:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3766 00000100:00080000:1.0:1775767265.063841:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767265.063847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767265.063851:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767265.063857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712832:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767265.063858:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767265.063877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767265.063878:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076713728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767265.063924:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.063944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350930000 x1862023076712960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.063957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076712960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.063965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.063969:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.063972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076712960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767265.063977:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767265.063995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350931880 x1862023076713088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064016:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076713088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064024:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064053:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767265.064063:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767265.064091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064101:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350932a00 x1862023076713216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076713216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064127:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767265.064148:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767265.064164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064173:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350930a80 x1862023076713344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064184:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076713344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064193:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064196:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767265.064205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767265.064221:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064231:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350930700 x1862023076713472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064255:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076713472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064270:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767265.064276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767265.064294:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064304:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350932680 x1862023076713600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064314:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076713600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064322:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064325:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064328:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767265.064334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767265.064349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767265.064358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767265/real 1775767265] req@ffffa04350931c00 x1862023076713728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767265.064369:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076713728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767265.064387:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767265.064390:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767265.064393:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076713728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767265.255101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767265.255105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767266.087161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767266.087220:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767266.279127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767266.279127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767267.111149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767267.111240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767267.303128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767267.303128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767268.135161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767268.135229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767268.327142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767268.327142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767269.159106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767269.159151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767269.351129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767269.351130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767270.183137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767270.183162:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183167:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767270.183171:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183180:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183188:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183193:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767270.183228:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076713856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767270.183248:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183264:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767270.183266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183269:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183273:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183277:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076713856:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767270.183287:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767270.183302:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076713984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767270.183312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767270.183316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183320:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183323:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183326:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00010000:00010000:1.0:1775767270.183335:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1775767270.183340:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767270.183341:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076714112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767270.183349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183352:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767270.183354:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183357:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183360:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183362:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767270.183379:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7e300 x1862023076713856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767270.183382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076714240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767270.183407:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767270.183411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767270.183414:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076713856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767270.183417:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183422:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183429:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183432:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767270.183438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076714368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767270.183439:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183447:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183454:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076713856:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767270.183454:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767270.183465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183471:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183475:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183479:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183484:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713984:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767270.183487:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767270.183505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076714496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767270.183515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767270.183517:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183520:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767270.183533:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7f480 x1862023076713984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767270.183541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183547:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183566:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767270.183570:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076713984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767270.183573:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183581:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775767270.183590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183595:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767270.183598:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076714624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767270.183602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076713984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767270.183615:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714112:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767270.183616:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183623:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767270.183624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00080000:2.0:1775767270.183630:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767270.183650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767270.183662:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076714752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767270.183669:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767270.183671:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767270.183672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767270.183677:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3771 00000100:00100000:0.0:1775767270.183679:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7e680 x1862023076714112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767270.183680:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767270.183686:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767270.183693:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767270.183701:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076714112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767270.183709:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076714880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767270.183710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767270.183733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767270.183774:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183783:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7c380 x1862023076714240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.183794:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076714240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.183816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183820:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767270.183831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767270.183860:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7dc00 x1862023076714368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.183876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076714368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.183882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076714368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767270.183900:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767270.183912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7c700 x1862023076714496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.183925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076714496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.183930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183932:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767270.183944:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767270.183959:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.183969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7c000 x1862023076714624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.183980:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076714624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.183989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.183992:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.183996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767270.184002:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767270.184021:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.184049:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7f100 x1862023076714752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.184058:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076714752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.184063:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.184065:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.184068:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767270.184074:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767270.184093:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767270.184112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767270/real 1775767270] req@ffffa0434aa7e300 x1862023076714880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767270.184125:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076714880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767270.184134:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767270.184136:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767270.184138:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076714880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767270.375105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767270.375107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767271.207091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767271.207162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767271.399111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767271.399201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767272.231095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767272.231180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767272.423119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767272.423121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767273.255126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767273.255151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767273.447137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767273.447137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767274.279134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767274.279230:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767274.471124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767274.471124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767275.303113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767275.303114:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303123:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767275.303127:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303135:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303145:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767275.303149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767275.303151:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767275.303193:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076715008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303221:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303225:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767275.303226:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303229:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303233:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767275.303237:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303244:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767275.303255:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076715136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767275.303264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076715008:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767275.303265:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303272:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767275.303275:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303278:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303280:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767275.303291:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303295:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767275.303306:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076715264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767275.303316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303318:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303322:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767275.303324:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767275.303334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767275.303344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076715392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303351:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303354:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767275.303355:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303360:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767275.303363:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303366:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767275.303381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7d500 x1862023076715008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767275.303381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076715520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303411:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767275.303419:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076715008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767275.303426:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767275.303431:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303440:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:0.0:1775767275.303441:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767275.303444:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767275.303449:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767275.303450:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303457:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767275.303459:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076715008:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767275.303469:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076715648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303479:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767275.303480:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303485:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767275.303488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715136:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767275.303491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767275.303505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076715776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303514:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767275.303516:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767275.303518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303520:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303522:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767275.303525:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767275.303539:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767275.303547:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076715904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767275.303555:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767275.303558:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7d880 x1862023076715136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767275.303559:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767275.303562:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767275.303572:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3776 00000100:00080000:2.0:1775767275.303580:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767275.303587:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076715136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767275.303587:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767275.303596:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767275.303607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.303614:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767275.303615:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076716032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767275.303620:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767275.303627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767275.303670:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.303682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7ea00 x1862023076715264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.303696:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076715264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.303733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.303737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.303741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767275.303747:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767275.303770:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.303781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7df80 x1862023076715392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.303823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076715392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.303834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.303837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.303840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767275.303847:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767275.303867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.303878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7fb80 x1862023076715520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.303890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076715520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.303899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.303903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.303906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076715520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767275.303912:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767275.303932:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.303942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7f800 x1862023076715648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.303954:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076715648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.303975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.303979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.303982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767275.304004:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767275.304024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.304057:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7ed80 x1862023076715776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.304071:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076715776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.304080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.304083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.304086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767275.304092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767275.304111:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.304121:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7ce00 x1862023076715904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.304133:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076715904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.304142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.304146:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.304157:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076715904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767275.304164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767275.304184:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767275.304195:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767275/real 1775767275] req@ffffa0434aa7d500 x1862023076716032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767275.304207:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076716032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767275.304216:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767275.304219:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767275.304223:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767275.495107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767275.495113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767276.327228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767276.327361:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767276.519143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767276.519148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767277.351126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767277.351177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767277.543111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767277.543111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767278.375149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767278.375239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767278.567131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767278.567131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767279.399125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767279.399168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767279.591204:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767279.591485:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767280.423142:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767280.423143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767280.423153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767280.423157:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423167:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3782 00010000:00010000:0.0:1775767280.423178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767280.423178:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423186:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767280.423225:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076716160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423249:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423264:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767280.423266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423270:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423274:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423278:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423287:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076716160:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767280.423287:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767280.423301:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076716288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423309:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767280.423313:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423315:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423320:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423323:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767280.423330:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076716416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423340:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767280.423341:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423343:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423345:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423347:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767280.423352:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423360:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767280.423369:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076716544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423376:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423378:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767280.423379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423383:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423385:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423388:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423391:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767280.423398:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076716672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767280.423407:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7e300 x1862023076716160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767280.423408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767280.423411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423418:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423424:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423429:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423433:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767280.423440:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076716160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767280.423442:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076716800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423470:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767280.423472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767280.423473:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767280.423478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767280.423480:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767280.423484:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423488:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767280.423490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076716160:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767280.423493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423500:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767280.423509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076716928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423515:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423517:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767280.423519:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423523:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00100000:0.0:1775767280.423524:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716288:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767280.423525:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423531:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767280.423542:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076717056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767280.423549:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767280.423552:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767280.423553:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767280.423556:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3782 00000100:00080000:2.0:1775767280.423559:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767280.423562:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767280.423566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767280.423569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7f100 x1862023076716288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767280.423585:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076717184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767280.423609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076716288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423616:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767280.423627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767280.423642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7c000 x1862023076716416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.423689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076716416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767280.423711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767280.423731:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7c700 x1862023076716544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.423769:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076716544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423774:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767280.423783:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767280.423798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7dc00 x1862023076716672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.423826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076716672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076716672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767280.423847:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767280.423868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7c380 x1862023076716800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.423889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076716800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767280.423921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767280.423940:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.423950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7e680 x1862023076716928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.423961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076716928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.423970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.423973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.423976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076716928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767280.423993:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767280.424007:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.424014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7f480 x1862023076717056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.424022:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076717056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.424045:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.424047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.424054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767280.424058:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767280.424073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767280.424080:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767280/real 1775767280] req@ffffa0434aa7e300 x1862023076717184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767280.424087:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076717184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767280.424093:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767280.424095:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767280.424097:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767280.615099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767280.615109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767281.447117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767281.447144:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767281.639127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767281.639127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767282.471102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767282.471176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767282.663103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767282.663105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767283.495115:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767283.495130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767283.687100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767283.687100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767284.519127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767284.519145:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767284.711108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767284.711108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767285.543125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767285.543198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767285.543200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543209:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767285.543214:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543229:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543243:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543252:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767285.543304:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076717312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767285.543336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543340:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767285.543342:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543347:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543352:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543357:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.543369:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767285.543376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076717312:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767285.543407:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076717440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767285.543421:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767285.543444:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543449:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543456:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767285.543459:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767285.543467:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.543474:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767285.543498:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076717568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767285.543511:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543515:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767285.543518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767285.543521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825ce00 x1862023076717312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767285.543527:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543536:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543545:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767285.543552:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076717312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767285.543553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767285.543573:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.543579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767285.543580:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076717696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767285.543587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076717312:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767285.543624:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543628:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767285.543631:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543639:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00100000:0.0:1775767285.543640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717440:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767285.543648:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543655:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.543670:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775767285.543671:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767285.543695:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076717824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767285.543702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825d880 x1862023076717440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767285.543753:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767285.543754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076717440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767285.543774:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767285.543777:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543783:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543789:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767285.543792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767285.543798:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767285.543800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:3.0:1775767285.543811:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767285.543812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767285.543828:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076717952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767285.543840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767285.543842:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543853:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767285.543859:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543864:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.543870:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543873:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.543877:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767285.543886:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076718080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767285.543897:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767285.543897:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543905:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767285.543914:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543920:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00100000:0.0:1775767285.543928:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825dc00 x1862023076717568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767285.543929:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.543935:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.543945:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767285.543960:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076718208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767285.543969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076717568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767285.543971:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767285.543981:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767285.543984:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767285.543993:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3787 00000100:00080000:3.0:1775767285.544001:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767285.544009:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767285.544014:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767285.544015:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767285.544053:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076718336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767285.544063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544071:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767285.544083:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767285.544115:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544125:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825d500 x1862023076717696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544134:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076717696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544149:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767285.544166:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767285.544181:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544188:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825f100 x1862023076717824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544196:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076717824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544201:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544204:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544206:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076717824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767285.544209:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767285.544219:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544225:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825e300 x1862023076717952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544231:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076717952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544237:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076717952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767285.544250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767285.544267:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544278:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825ed80 x1862023076718080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544290:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076718080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544299:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544303:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544306:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767285.544313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767285.544330:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544338:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825c000 x1862023076718208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544345:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076718208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544351:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544353:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544355:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767285.544359:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767285.544369:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767285.544381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767285/real 1775767285] req@ffffa0435825e680 x1862023076718336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767285.544389:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076718336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767285.544394:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767285.544396:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767285.544398:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076718336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767285.735170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767285.735175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767286.567101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767286.567203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767286.759162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767286.759163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767287.591117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767287.591173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767287.783118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767287.783118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767288.615118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767288.615173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767288.807116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767288.807116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767289.639124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767289.639190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767289.831101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767289.831116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:3.0:1775767290.663095:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767290.663103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767290.663103:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767290.663106:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663116:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663125:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663131:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767290.663165:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076718464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767290.663203:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663206:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767290.663208:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663211:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663214:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663217:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663225:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767290.663236:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076718592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767290.663242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076718464:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767290.663246:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663252:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767290.663253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663256:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663258:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663261:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663263:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767290.663272:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076718720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767290.663278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663280:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767290.663282:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663284:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663292:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663295:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663298:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767290.663304:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076718848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767290.663311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663313:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767290.663314:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663316:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663318:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767290.663323:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767290.663325:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663329:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767290.663336:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076718976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767290.663342:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663345:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767290.663346:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663349:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663351:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767290.663364:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076719104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767290.663370:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825ea00 x1862023076718464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767290.663406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767290.663410:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663413:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663416:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767290.663418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076718464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767290.663421:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663428:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767290.663439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076719232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767290.663440:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767290.663450:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767290.663451:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767290.663457:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767290.663458:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767290.663461:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663463:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663465:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663467:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767290.663469:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076718464:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767290.663479:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076719360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767290.663488:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767290.663491:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767290.663493:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767290.663497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718592:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767290.663499:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3792 00000100:00080000:3.0:1775767290.663511:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767290.663516:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767290.663523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767290.663527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767290.663541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076719488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767290.663542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825e680 x1862023076718592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076718592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663595:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767290.663605:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767290.663625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663632:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825c000 x1862023076718720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663640:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076718720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767290.663656:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767290.663669:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663675:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825ed80 x1862023076718848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663690:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076718848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767290.663724:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718976:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775767290.663737:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1775767290.663757:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825e300 x1862023076718976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076718976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663801:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076718976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767290.663809:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767290.663830:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825f100 x1862023076719104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076719104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767290.663870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767290.663885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663905:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825d500 x1862023076719232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076719232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767290.663934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767290.663954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.663970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825dc00 x1862023076719360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.663984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076719360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.663993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.663997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.663999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767290.664047:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767290.664070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767290.664085:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767290/real 1775767290] req@ffffa0435825d880 x1862023076719488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767290.664101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076719488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767290.664117:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767290.664122:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767290.664126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076719488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767290.855084:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767290.855089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767291.687154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767291.687221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767291.879135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767291.879135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767292.711121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767292.711191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767292.903112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767292.903112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767293.735117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767293.735163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767293.927107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767293.927127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767294.759112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767294.759195:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767294.951142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767294.951155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767295.783119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767295.783158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767295.783175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767295.783188:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783198:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783209:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783217:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767295.783255:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076719616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783293:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767295.783300:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783306:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783317:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783325:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076719616:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767295.783327:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767295.783342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076719744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783367:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767295.783368:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783372:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783376:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783378:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767295.783384:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767295.783385:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767295.783397:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076719872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767295.783408:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783410:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783412:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783415:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767295.783419:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767295.783419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349640e00 x1862023076719616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767295.783429:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076720000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767295.783446:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783448:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783450:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783452:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767295.783461:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767295.783471:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076719616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767295.783472:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076720128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783505:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767295.783506:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767295.783526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767295.783529:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783533:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783539:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767295.783541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767295.783545:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783547:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076719616:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767295.783552:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767295.783564:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076720256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767295.783572:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767295.783573:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076719744:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767295.783580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767295.783582:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783586:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783588:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767295.783591:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767295.783594:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767295.783602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767295.783606:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076720384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767295.783615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349641c00 x1862023076719744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767295.783620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767295.783629:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767295.783639:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767295.783642:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076719744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767295.783648:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:2.0:1775767295.783657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767295.783658:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767295.783662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767295.783668:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076719744:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767295.783679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767295.783682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076719872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767295.783700:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076720512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767295.783705:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767295.783714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767295.783716:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349641180 x1862023076719872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767295.783718:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767295.783721:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767295.783729:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3797 00000100:00080000:0.0:1775767295.783735:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767295.783738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076719872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767295.783742:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767295.783749:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767295.783760:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076720640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767295.783762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.783770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.783775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076719872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767295.783781:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767295.783798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783805:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349642680 x1862023076720000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.783812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076720000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.783818:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.783821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.783824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767295.783830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767295.783848:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349640700 x1862023076720128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.783871:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076720128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.783880:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.783884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.783892:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767295.783907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767295.783923:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349640a80 x1862023076720256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.783938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076720256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.783944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.783946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.783948:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767295.783951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767295.783972:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.783978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349641880 x1862023076720384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.783985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076720384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.783990:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.783992:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.783994:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767295.783997:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767295.784006:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.784019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349643800 x1862023076720512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.784044:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076720512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.784049:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.784052:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.784054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767295.784058:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767295.784070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767295.784077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767295/real 1775767295] req@ffffa04349643b80 x1862023076720640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767295.784084:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076720640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767295.784089:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767295.784090:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767295.784092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076720640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767295.975121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767295.975127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767296.807162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767296.807239:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767296.999141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767296.999149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767297.831154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767297.831181:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767298.023108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767298.023111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767298.855106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767298.855152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767299.047125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767299.047178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767299.879098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767299.879143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767300.071106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767300.071111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767300.903111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767300.903142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767300.903155:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767300.903159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767300.903165:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903176:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903183:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767300.903222:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076720768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903256:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767300.903258:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903263:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903268:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767300.903282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076720768:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767300.903283:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767300.903307:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076720896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767300.903347:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903352:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903358:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903363:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767300.903368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720896:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767300.903370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767300.903389:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076721024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767300.903411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903416:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903421:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767300.903423:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767300.903430:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903437:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767300.903453:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076721152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903487:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767300.903489:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903493:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903506:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903511:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903516:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767300.903529:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076721280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903539:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767300.903545:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903549:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903553:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903557:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903561:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767300.903572:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076721408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767300.903586:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903589:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903593:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903597:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767300.903612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076721536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903633:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767300.903638:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903642:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903646:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903650:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903655:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767300.903666:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076721664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767300.903675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767300.903678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767300.903680:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767300.903684:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3802 00000100:00080000:1.0:1775767300.903687:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767300.903691:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767300.903695:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767300.903704:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076721792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767300.903808:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767300.903839:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767300.903859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767300.903878:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767300.903896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767300.903912:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767301.095129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767301.095138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767301.927106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767301.927141:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767302.119101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767302.119128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767302.951096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767302.951118:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767303.143139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767303.143139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767303.975152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767303.975453:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775767303.975473:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775767303.975537:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767303.975558:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975675:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975693:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975701:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975710:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975718:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975726:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975734:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767303.975742:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775767303.975872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350930a80 x1862023076721152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.975907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076721152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.975919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.975924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.975929:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767303.975952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350932a00 x1862023076721280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.975967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076721280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.975976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.975981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.975985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767303.975993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350931880 x1862023076721408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076721408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767303.976071:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350930000 x1862023076721536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976086:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076721536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976095:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976099:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976102:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767303.976116:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350930e00 x1862023076721664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976129:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076721664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976138:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767303.976152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350932300 x1862023076721792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976164:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076721792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976171:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976175:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976178:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076721792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767303.976186:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350930700 x1862023076721024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076721024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976208:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976211:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976214:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076721024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767303.976229:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350932680 x1862023076720896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976248:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076720896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976257:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976261:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976264:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076720896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767303.976271:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767300/real 1775767303] req@ffffa04350931c00 x1862023076720768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767303.976284:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076720768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767303.976292:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767303.976295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767303.976298:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076720768:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767304.167123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767304.167123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767304.999150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767304.999201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767305.191149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767305.191149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767306.023118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767306.023179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767306.023192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.023200:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767306.023205:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023216:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023229:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.023239:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767306.023278:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076721920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023304:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.023318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767306.023321:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023327:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023332:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.023338:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023348:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767306.023354:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076721920:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767306.023383:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076722048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.023400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767306.023402:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023407:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023413:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.023432:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767306.023458:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076722176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767306.023476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767306.023482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767306.023485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023492:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023498:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.023504:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023512:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767306.023529:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076722304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767306.023544:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825ce00 x1862023076721920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767306.023547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767306.023553:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023571:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023578:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767306.023580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076721920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767306.023589:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023601:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775767306.023620:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767306.023623:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076722432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.023641:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767306.023644:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767306.023646:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767306.023653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076721920:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767306.023655:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023669:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.023677:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023686:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767306.023707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767306.023708:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076722560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.023735:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.023738:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767306.023740:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.023745:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.023750:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767306.023756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767306.023762:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.023770:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767306.023782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825d880 x1862023076722048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767306.023792:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076722688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767306.023805:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076722048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767306.023841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767306.023843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767306.023851:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767306.023852:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767306.023858:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767306.023860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767306.023873:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00100000:0.0:1775767306.023879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767306.023904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.023916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825dc00 x1862023076722176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.023929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076722176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.023937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.023941:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.023944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767306.023950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767306.023980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.023993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825d500 x1862023076722304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.024006:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076722304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.024014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.024017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.024020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767306.024054:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767306.024076:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.024088:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825f100 x1862023076722432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.024100:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076722432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.024109:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.024112:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.024116:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076722432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767306.024140:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767306.024164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.024177:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825e300 x1862023076722560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.024205:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076722560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.024214:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.024217:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.024221:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767306.024228:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767306.024251:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.024263:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825ed80 x1862023076722688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.024272:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076722688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.024278:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.024280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.024282:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767306.024362:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.024370:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.024378:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767306.024396:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076722816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767306.024436:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767306.024440:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767306.024442:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767306.024446:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3807 00000100:00080000:3.0:1775767306.024451:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767306.024455:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767306.024460:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767306.024472:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076722944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767306.025219:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767306.025261:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.025280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825c000 x1862023076722816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.025295:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076722816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.025429:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.025434:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.025438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767306.025447:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767306.025473:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767306.025486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767306/real 1775767306] req@ffffa0435825e680 x1862023076722944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767306.025512:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076722944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767306.025520:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767306.025523:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767306.025527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076722944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767306.215105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767306.215105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767307.047121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767307.047159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767307.239122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767307.239122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767308.071110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767308.071163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767308.263111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767308.263195:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767309.095127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767309.096065:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767309.287118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767309.287128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767310.119121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767310.119187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767310.311108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767310.311112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767311.143116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767311.143121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767311.143153:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143163:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143173:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767311.143180:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767311.143216:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076723072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767311.143253:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767311.143259:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143264:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143268:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767311.143273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143282:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767311.143291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076723072:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767311.143298:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076723200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767311.143310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767311.143315:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143320:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143324:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767311.143328:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143333:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767311.143345:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076723328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767311.143359:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767311.143388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767311.143394:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143400:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143406:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767311.143410:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767311.143428:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825ea00 x1862023076723072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767311.143432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076723456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767311.143458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143466:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767311.143469:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143474:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143478:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767311.143480:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076723072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767311.143486:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775767311.143497:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143504:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143507:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076723072:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767311.143512:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076723584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767311.143527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767311.143533:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767311.143541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767311.143546:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143558:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767311.143565:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767311.143566:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143573:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767311.143577:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825e680 x1862023076723200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767311.143585:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076723712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767311.143592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076723200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767311.143599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143603:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767311.143604:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767311.143605:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767311.143611:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767311.143612:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143616:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767311.143617:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767311.143623:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767311.143627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723328:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767311.143628:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767311.143640:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076723840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767311.143647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767311.143648:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767311.143653:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767311.143654:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143658:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00100000:0.0:1775767311.143661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825c000 x1862023076723328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767311.143663:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767311.143670:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143678:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767311.143682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076723328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767311.143688:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076723968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767311.143694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143699:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767311.143715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723456:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767311.143718:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767311.143721:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767311.143723:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767311.143727:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3812 00000100:00080000:3.0:1775767311.143731:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767311.143734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767311.143738:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767311.143743:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767311.143746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825ed80 x1862023076723456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767311.143753:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076724096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767311.143791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076723456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.143798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143801:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143803:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767311.143809:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723584:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775767311.143823:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:0.0:1775767311.143843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767311.143855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825e300 x1862023076723584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767311.143867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076723584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.143876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076723584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767311.143897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767311.143912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767311.143919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825f100 x1862023076723712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767311.143927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076723712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.143942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767311.143950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767311.143963:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767311.143969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825d500 x1862023076723840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767311.143976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076723840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.143981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.143984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.143985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767311.143988:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767311.143997:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767311.144002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825dc00 x1862023076723968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767311.144009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076723968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.144014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.144016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.144022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076723968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767311.144043:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767311.144058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767311.144065:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767311/real 1775767311] req@ffffa0435825d880 x1862023076724096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767311.144073:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076724096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767311.144078:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767311.144080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767311.144082:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767311.335138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767311.335139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767312.167129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767312.167184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767312.359104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767312.359105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767313.191165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767313.191222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767313.383157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767313.383163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767314.215165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767314.215177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767314.407106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767314.407152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767315.239166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767315.239169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767315.431131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767315.431163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775767316.263116:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263125:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767316.263129:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263138:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263148:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767316.263154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767316.263154:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767316.263192:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076724224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767316.263216:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263219:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767316.263220:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263223:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263227:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263245:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076724224:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775767316.263254:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767316.263272:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076724352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767316.263285:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263290:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767316.263292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263296:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263301:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263305:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263310:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767316.263338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767316.263349:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076724480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767316.263358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767316.263362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263365:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263372:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825ce00 x1862023076724224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767316.263379:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767316.263392:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076724608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767316.263397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076724224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767316.263404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767316.263410:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767316.263416:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767316.263418:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767316.263419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767316.263424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076724224:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767316.263426:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263434:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263443:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263448:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767316.263452:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767316.263460:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076724736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767316.263472:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263474:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767316.263476:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263479:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263482:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263487:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767316.263488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767316.263504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825d880 x1862023076724352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767316.263505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076724864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767316.263536:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767316.263538:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076724352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767316.263545:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767316.263550:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767316.263552:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263558:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767316.263558:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00100000:0.0:1775767316.263563:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724352:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767316.263566:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767316.263570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767316.263574:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263587:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767316.263588:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263598:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825dc00 x1862023076724480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767316.263611:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076724480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767316.263611:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076724992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767316.263635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767316.263639:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767316.263641:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767316.263645:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767316.263647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767316.263648:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263656:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00100000:0.0:1775767316.263659:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724608:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767316.263661:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263666:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263670:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767316.263675:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767316.263681:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076725120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767316.263685:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825d500 x1862023076724608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767316.263699:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767316.263702:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767316.263703:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767316.263706:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3817 00000100:00080000:3.0:1775767316.263711:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767316.263717:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767316.263723:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767316.263729:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076724608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767316.263732:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076725248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767316.263753:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767316.263775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767316.263793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825f100 x1862023076724736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767316.263810:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076724736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767316.263817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076724736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767316.263826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767316.263838:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263844:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825e300 x1862023076724864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1775767316.263848:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767316.263853:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076724864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767316.263867:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263872:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767316.263875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767316.263887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825ed80 x1862023076724992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767316.263900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076724992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767316.263905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076724992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767316.263912:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076725120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767316.263921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825c000 x1862023076725120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767316.263932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076725120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767316.263937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076725120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767316.263948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076725248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767316.263960:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767316.263966:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767316/real 1775767316] req@ffffa0435825e680 x1862023076725248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767316.263973:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076725248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767316.263978:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767316.263980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767316.263982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076725248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767316.455110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767316.455163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767317.287182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767317.287235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767317.479183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767317.479186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767318.311152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767318.311193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767318.503102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767318.503126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775767319.335196:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767319.335214:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767319.527120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767319.527120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767320.359146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767320.359244:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767320.551128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767320.551128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767321.383167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767321.383197:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383206:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767321.383211:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383222:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3823 00010000:00010000:3.0:1775767321.383222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767321.383236:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383243:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767321.383318:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076725376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767321.383346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383349:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767321.383351:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383354:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383357:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383360:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383368:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767321.383379:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076725504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767321.383399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076725376:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767321.383405:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767321.383410:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383414:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383417:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383420:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383423:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767321.383432:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076725632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767321.383438:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383440:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767321.383441:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383445:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383447:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383450:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767321.383457:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076725760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767321.383463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767321.383465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383468:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383476:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383479:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383482:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767321.383491:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076725888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767321.383498:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767321.383500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767321.383501:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767321.383504:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383508:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383510:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383513:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383516:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767321.383524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076726016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767321.383530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383533:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767321.383534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383536:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383539:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383541:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383544:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767321.383552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7d500 x1862023076725376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767321.383554:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076726144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767321.383590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076725376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767321.383596:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383601:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767321.383604:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383611:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383616:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767321.383617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767321.383619:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383625:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767321.383629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767321.383636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076726272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767321.383638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076725376:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767321.383652:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767321.383654:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767321.383656:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767321.383659:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3823 00000100:00080000:2.0:1775767321.383661:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767321.383663:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767321.383665:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767321.383672:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076726400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767321.383677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767321.383710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.383747:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7ce00 x1862023076725504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.383760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076725504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.383770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.383775:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.383779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767321.383786:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767321.383810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.383822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7ed80 x1862023076725632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.383835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076725632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.383844:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.383848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.383851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767321.383859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767321.383877:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.383888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7f800 x1862023076725760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.383900:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076725760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.383916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.383921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.383924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767321.383931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767321.383967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.383979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7fb80 x1862023076725888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.383991:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076725888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.384001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.384004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.384008:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076725888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767321.384014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767321.384061:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.384075:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7df80 x1862023076726016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.384089:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076726016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.384098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.384102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.384114:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767321.384122:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767321.384148:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.384159:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7ea00 x1862023076726144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.384172:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076726144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.384181:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.384184:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.384188:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767321.384194:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767321.384214:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.384225:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7d880 x1862023076726272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.384237:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076726272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.384245:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.384249:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.384253:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767321.384308:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767321.384330:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767321.384349:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767321/real 1775767321] req@ffffa0434aa7d500 x1862023076726400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767321.384362:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076726400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767321.384370:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767321.384374:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767321.384377:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076726400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767321.575154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767321.575154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767322.407143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767322.407255:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767322.599213:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767322.599213:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767323.431158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767323.431217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767323.623096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767323.623141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767324.455095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767324.455178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767324.647140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767324.647140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767325.479136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767325.479184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767325.671183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767325.671183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767326.503135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767326.503193:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503200:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767326.503204:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503213:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503237:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503244:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767326.503271:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767326.503277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076726528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503302:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767326.503304:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503308:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503312:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503316:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767326.503328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076726528:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767326.503337:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076726656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503349:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767326.503350:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503353:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503364:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503367:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503372:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767326.503383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076726784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503391:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767326.503396:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767326.503397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767326.503400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503405:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503408:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503411:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767326.503424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076726912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503430:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503432:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767326.503434:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503438:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00100000:3.0:1775767326.503438:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350931c00 x1862023076726528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767326.503440:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503445:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767326.503461:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076727040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767326.503466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076726528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767326.503486:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767326.503489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767326.503491:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767326.503493:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767326.503494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076726528:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767326.503499:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503503:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503505:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503508:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767326.503514:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076727168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503519:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503522:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767326.503523:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503525:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503526:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503528:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503530:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767326.503535:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076727296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767326.503542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767326.503543:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503545:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503551:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503555:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503559:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767326.503567:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076727424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767326.503574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767326.503575:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767326.503579:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767326.503582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767326.503586:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3828 00000100:00080000:1.0:1775767326.503589:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767326.503591:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767326.503595:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767326.503603:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767326.503606:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076727552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767326.503617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350932680 x1862023076726656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.503626:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076726656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503633:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.503737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.503740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767326.503746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767326.503766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.503785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350930700 x1862023076726784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.503795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076726784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503802:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.503856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.503860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767326.503865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767326.503881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.503888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350932300 x1862023076726912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.503904:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076726912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.503911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.503913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076726912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767326.503917:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767326.503928:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.503934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350930e00 x1862023076727040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.503941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076727040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.503954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.503956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767326.503960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767326.503974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.503980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350930000 x1862023076727168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.503987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076727168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.503992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.503995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.503998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767326.504003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767326.504022:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.504059:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350931880 x1862023076727296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.504076:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076727296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.504085:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.504089:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.504099:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767326.504121:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767326.504144:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.504155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350932a00 x1862023076727424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.504168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076727424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.504176:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.504179:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.504183:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767326.504189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767326.504210:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767326.504222:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767326/real 1775767326] req@ffffa04350930a80 x1862023076727552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767326.504251:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076727552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767326.504257:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767326.504259:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767326.504262:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076727552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767326.695140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767326.695140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767327.527131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767327.527148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767327.719105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767327.719105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767328.551127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767328.551163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767328.743125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767328.743126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767329.575114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767329.575143:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767329.767128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767329.767128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767330.599143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767330.599175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767330.791128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767330.791128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767331.623123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767331.623133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623141:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767331.623145:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623156:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3833 00010000:00010000:3.0:1775767331.623160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767331.623166:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623172:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767331.623224:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076727680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767331.623252:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623256:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623261:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623275:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767331.623278:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076727680:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767331.623289:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076727808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767331.623316:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623320:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623323:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623332:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767331.623333:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767331.623341:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076727936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767331.623356:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623359:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623362:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623366:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767331.623378:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7e300 x1862023076727680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767331.623379:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076728064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623392:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767331.623393:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623397:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00100000:3.0:1775767331.623397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076727680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767331.623400:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623403:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623407:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767331.623408:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623412:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623416:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076727680:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767331.623423:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076728192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767331.623435:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727808:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767331.623437:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623441:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623444:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623448:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623453:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767331.623456:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767331.623462:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076728320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767331.623478:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7f480 x1862023076727808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767331.623479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767331.623488:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767331.623490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076727808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767331.623492:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623495:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623498:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767331.623500:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767331.623503:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767331.623503:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623506:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767331.623512:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767331.623513:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076728448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767331.623521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767331.623526:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767331.623534:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:2.0:1775767331.623538:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767331.623542:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767331.623542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623547:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767331.623552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7e680 x1862023076727936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767331.623557:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076728576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767331.623565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076727936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767331.623567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767331.623570:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767331.623572:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767331.623574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767331.623576:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3833 00000100:00080000:3.0:1775767331.623577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767331.623579:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767331.623580:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076727936:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767331.623583:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767331.623587:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767331.623587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767331.623595:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076728704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767331.623603:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623612:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7c380 x1862023076728064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623624:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076728064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623633:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767331.623644:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767331.623659:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7dc00 x1862023076728192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076728192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623690:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623692:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623694:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767331.623705:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767331.623718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7c700 x1862023076728320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623731:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076728320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623739:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623742:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767331.623751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767331.623769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7c000 x1862023076728448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076728448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767331.623825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767331.623840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7f100 x1862023076728576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076728576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767331.623865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767331.623875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767331.623880:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767331/real 1775767331] req@ffffa0434aa7e300 x1862023076728704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767331.623887:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076728704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767331.623891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767331.623893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767331.623899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076728704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767331.815112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767331.815113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767332.647119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767332.647155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767332.839125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767332.839125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767333.671149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767333.671177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767333.863107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767333.863107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767334.695143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767334.695178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767334.887124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767334.887130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767335.719115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767335.719167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767335.911107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767335.911108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767336.743123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767336.743142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767336.743154:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743166:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3838 00010000:00010000:3.0:1775767336.743171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767336.743177:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743184:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767336.743222:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076728832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767336.743244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767336.743250:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743254:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743259:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743275:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767336.743277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076728832:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767336.743292:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076728960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767336.743306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743310:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767336.743312:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743316:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743334:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743337:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767336.743353:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076729088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767336.743360:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767336.743367:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743369:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767336.743371:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743374:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743377:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743380:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743383:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767336.743391:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076729216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767336.743397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743399:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767336.743401:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743403:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743406:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767336.743411:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350930a80 x1862023076728832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767336.743418:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767336.743438:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076729344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767336.743438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076728832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743473:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767336.743474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767336.743482:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767336.743483:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767336.743489:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076728832:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767336.743490:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743496:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743500:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743504:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767336.743512:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076729472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767336.743515:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728960:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767336.743527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767336.743531:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743535:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743537:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743540:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743543:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767336.743551:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767336.743552:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076729600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767336.743561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767336.743565:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767336.743568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350932a00 x1862023076728960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767336.743570:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00080000:1.0:1775767336.743581:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767336.743585:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076728960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767336.743586:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743593:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767336.743597:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767336.743605:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076729728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767336.743609:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076728960:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767336.743614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767336.743620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767336.743623:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767336.743628:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3838 00000100:00100000:3.0:1775767336.743629:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729088:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767336.743631:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767336.743637:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767336.743641:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767336.743650:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076729856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767336.743658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350931880 x1862023076729088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076729088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743683:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743688:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767336.743691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767336.743702:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350930000 x1862023076729216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076729216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743722:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767336.743725:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767336.743734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350930e00 x1862023076729344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743753:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076729344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743763:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076729344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767336.743776:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767336.743788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350932300 x1862023076729472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076729472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767336.743822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767336.743833:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743839:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350930700 x1862023076729600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743846:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076729600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743859:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743861:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767336.743867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767336.743879:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350932680 x1862023076729728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076729728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767336.743904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767336.743913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767336.743917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767336/real 1775767336] req@ffffa04350931c00 x1862023076729856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767336.743924:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076729856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767336.743929:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767336.743931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767336.743936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076729856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767336.935122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767336.935142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767337.767140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767337.767194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767337.959155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767337.959155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767338.791141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767338.791177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767338.983118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767338.983119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767339.815096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767339.815171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767340.007148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767340.007153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767340.839124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767340.839149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767341.031121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767341.031122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767341.863120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767341.863187:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863196:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767341.863201:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863211:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3843 00010000:00010000:2.0:1775767341.863235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767341.863256:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863264:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767341.863309:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076729984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863344:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863347:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767341.863348:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863351:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863354:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863357:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863367:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767341.863377:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076730112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863385:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767341.863390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863395:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863397:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767341.863397:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076729984:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767341.863401:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767341.863415:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076730240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863421:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863430:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767341.863431:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863434:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863438:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863441:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863445:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767341.863454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076730368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863461:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767341.863465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863467:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863477:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863479:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863482:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767341.863488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767341.863492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076730496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863498:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767341.863503:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863507:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863514:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863517:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863526:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767341.863536:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076730624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863548:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767341.863551:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350931c00 x1862023076729984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767341.863556:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767341.863561:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863570:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863575:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767341.863576:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076729984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767341.863586:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767341.863590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863595:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076729984:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767341.863601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767341.863615:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076730752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767341.863622:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863626:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767341.863627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730112:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767341.863642:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767341.863650:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863653:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863656:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767341.863661:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767341.863670:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076730880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767341.863678:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767341.863684:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767341.863687:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767341.863694:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767341.863699:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350932680 x1862023076730112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767341.863701:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3843 00000100:00080000:1.0:1775767341.863709:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767341.863718:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767341.863723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076730112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767341.863725:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767341.863738:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863742:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767341.863749:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076731008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767341.863755:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767341.863785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.863794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350930700 x1862023076730240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.863803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076730240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.863809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767341.863817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767341.863828:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.863834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350932300 x1862023076730368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.863842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076730368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.863847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767341.863855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767341.863865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.863871:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350930e00 x1862023076730496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.863889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076730496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.863896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863900:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076730496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767341.863903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767341.863920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.863941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350930000 x1862023076730624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.863959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076730624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.863967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.863970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.863973:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767341.863979:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767341.864001:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.864012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350931880 x1862023076730752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.864063:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076730752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.864099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.864102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.864105:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767341.864122:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767341.864140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.864148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350932a00 x1862023076730880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.864156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076730880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.864161:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.864164:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.864166:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076730880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767341.864169:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767341.864179:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767341.864184:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767341/real 1775767341] req@ffffa04350930a80 x1862023076731008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767341.864191:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076731008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767341.864197:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767341.864199:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767341.864206:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767342.055213:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767342.055265:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767342.887139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767342.887189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767343.079176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767343.079199:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767343.911155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767343.911221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767344.103123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767344.103124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767344.935118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767344.935183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767345.127171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767345.127177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767345.959181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767345.959240:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767346.151197:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767346.151197:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767346.983170:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983181:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767346.983187:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767346.983193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767346.983199:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983215:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767346.983224:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767346.983225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767346.983278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076731136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767346.983334:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983359:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767346.983362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983371:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00100000:3.0:1775767346.983372:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076731136:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767346.983382:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767346.983391:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767346.983421:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767346.983442:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076731264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767346.983455:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767346.983473:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983478:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767346.983480:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983488:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983494:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767346.983499:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7d500 x1862023076731136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.983505:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767346.983529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767346.983532:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076731136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.983546:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767346.983554:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076731392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767346.983554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.983559:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076731136:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767346.983595:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983599:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767346.983601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983605:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983608:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767346.983612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731264:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767346.983614:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767346.983631:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767346.983640:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767346.983647:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076731520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767346.983657:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7d880 x1862023076731264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.983694:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767346.983698:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076731264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.983699:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767346.983704:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983720:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983730:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767346.983730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.983743:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767346.983744:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767346.983756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731264:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767346.983757:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767346.983767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767346.983772:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076731648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767346.983781:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983784:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767346.983786:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767346.983793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767346.983798:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983801:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767346.983804:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767346.983808:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767346.983821:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076731776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767346.983827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7ea00 x1862023076731392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.983840:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983852:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767346.983859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076731392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.983863:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983878:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:3.0:1775767346.983882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767346.983884:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767346.983889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767346.983893:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767346.983899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731392:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767346.983904:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767346.983920:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076731904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767346.983923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731520:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767346.983955:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.983958:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767346.983960:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.983964:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00080000:2.0:1775767346.983969:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767346.983974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767346.983978:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767346.983984:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767346.983997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7df80 x1862023076731520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767346.983998:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076732032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767346.984028:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767346.984085:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767346.984088:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767346.984097:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3848 00000100:00100000:3.0:1775767346.984098:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076731520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767346.984106:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767346.984114:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767346.984119:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767346.984126:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984129:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731520:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775767346.984130:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767346.984135:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767346.984157:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076732160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767346.984167:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767346.984182:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7fb80 x1862023076731648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767346.984196:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076731648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.984208:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.984211:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076731648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767346.984217:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767346.984232:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767346.984239:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7f800 x1862023076731776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767346.984247:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076731776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.984252:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.984254:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984256:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767346.984268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767346.984288:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767346.984294:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7ed80 x1862023076731904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767346.984301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076731904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.984306:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.984307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984314:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076731904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767346.984318:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767346.984331:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767346.984337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7ce00 x1862023076732032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767346.984344:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076732032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.984349:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.984351:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984352:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767346.984356:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767346.984366:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767346.984371:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767346/real 1775767346] req@ffffa0434aa7ea00 x1862023076732160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767346.984378:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076732160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767346.984383:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767346.984385:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767346.984387:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767347.175172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767347.175191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775767348.007154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767348.007207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767348.199146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767348.199167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767349.031128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767349.031260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767349.223108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767349.223164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767350.055130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767350.055169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767350.247140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767350.247140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767351.079108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767351.079163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767351.271101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767351.271101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767352.103100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767352.103107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767352.103108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767352.103113:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103122:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103130:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103137:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767352.103160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767352.103170:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076732288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767352.103189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767352.103194:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103198:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103201:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103203:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767352.103219:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076732288:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767352.103225:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076732416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767352.103251:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767352.103257:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103262:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103278:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767352.103284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767352.103290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076732544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767352.103298:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767352.103302:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103304:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103306:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103309:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767352.103321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076732672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767352.103325:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7d880 x1862023076732288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767352.103333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767352.103342:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103349:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103356:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767352.103357:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076732288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767352.103362:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103368:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767352.103377:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767352.103382:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076732800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767352.103385:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103394:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076732288:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767352.103396:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767352.103403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767352.103406:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103409:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103413:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103418:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767352.103423:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767352.103431:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076732928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767352.103443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767352.103450:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767352.103454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767352.103458:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103464:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103469:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103474:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767352.103474:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7d500 x1862023076732416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767352.103487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076733056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767352.103497:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076732416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767352.103512:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767352.103518:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767352.103519:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767352.103523:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103527:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767352.103528:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767352.103532:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103537:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767352.103538:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767352.103548:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076733184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767352.103561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767352.103564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767352.103566:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767352.103567:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732544:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767352.103571:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3853 00000100:00080000:2.0:1775767352.103575:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767352.103578:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767352.103581:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767352.103589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076733312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767352.103616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.103631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7e300 x1862023076732544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.103646:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076732544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.103655:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.103658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767352.103669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767352.103688:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.103697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7f100 x1862023076732672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.103710:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076732672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.103718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.103721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767352.103731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767352.103750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.103761:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7c000 x1862023076732800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.103774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076732800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.103783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.103786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076732800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767352.103808:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767352.103828:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.103838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7c700 x1862023076732928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.103852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076732928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.103862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.103866:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076732928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767352.103876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767352.103896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.103907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7dc00 x1862023076733056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.103919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076733056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.103933:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.103936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.103949:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767352.103955:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767352.103975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.104001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7c380 x1862023076733184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.104013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076733184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.104022:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.104047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.104051:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767352.104058:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767352.104080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767352.104092:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767352/real 1775767352] req@ffffa0434aa7e680 x1862023076733312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767352.104105:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076733312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767352.104114:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767352.104116:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767352.104120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076733312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767352.295113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767352.295113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767353.127112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767353.127156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767353.319105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767353.319106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767354.151102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767354.151123:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767354.343107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767354.343107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767355.175128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767355.175172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767355.367098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767355.367102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767356.199131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767356.199158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767356.391123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767356.391128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767357.223114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767357.223149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767357.223160:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775767357.223172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767357.223188:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223199:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223205:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767357.223241:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076733440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767357.223271:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767357.223277:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767357.223282:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223288:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223293:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767357.223304:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767357.223304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076733440:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767357.223324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076733568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767357.223337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767357.223341:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767357.223344:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223349:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223352:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767357.223358:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767357.223364:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767357.223373:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076733696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767357.223398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767357.223399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350930a80 x1862023076733440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767357.223414:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767357.223422:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00100000:3.0:1775767357.223426:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076733440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223428:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223438:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767357.223439:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.223446:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767357.223447:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767357.223450:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076733440:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767357.223468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767357.223477:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076733824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767357.223487:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767357.223493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767357.223500:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767357.223505:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00100000:3.0:1775767357.223509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350932a00 x1862023076733568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223510:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223519:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767357.223528:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076733568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767357.223529:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767357.223542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.223546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767357.223551:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076733952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767357.223553:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767357.223563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733696:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767357.223576:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767357.223582:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767357.223587:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767357.223591:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00100000:3.0:1775767357.223597:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350931880 x1862023076733696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223608:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767357.223611:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076733696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223619:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767357.223624:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767357.223628:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767357.223632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767357.223639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767357.223649:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076734080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767357.223659:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733824:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767357.223664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223668:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767357.223671:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767357.223677:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223682:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223687:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767357.223693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767357.223695:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767357.223707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350930000 x1862023076733824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767357.223713:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076734208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767357.223722:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076733824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767357.223734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.223740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767357.223741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767357.223745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733824:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767357.223746:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767357.223754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733952:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767357.223755:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223764:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767357.223769:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767357.223776:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767357.223779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767357.223789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350930e00 x1862023076733952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767357.223796:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076734336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767357.223803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076733952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767357.223820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.223823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767357.223829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076733952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767357.223845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767357.223847:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767357.223852:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775767357.223860:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767357.223865:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767357.223868:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350932300 x1862023076734080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223875:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3858 00000100:00080000:1.0:1775767357.223884:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767357.223884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076734080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767357.223895:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767357.223896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.223904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767357.223906:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767357.223907:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767357.223916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767357.223929:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076734464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767357.223967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767357.223984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350930700 x1862023076734208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767357.224003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076734208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767357.224009:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.224011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767357.224013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767357.224023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767357.224058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767357.224070:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350932680 x1862023076734336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767357.224083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076734336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767357.224092:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.224096:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767357.224100:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767357.224107:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767357.224126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767357.224134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767357/real 1775767357] req@ffffa04350931c00 x1862023076734464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767357.224143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076734464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767357.224153:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767357.224155:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767357.224157:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076734464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767357.415081:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767357.415082:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767358.247127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767358.247146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767358.439099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767358.439099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767359.271106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767359.271160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767359.463136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767359.463136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767360.295115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767360.295170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767360.487098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767360.487107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767361.319105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767361.319176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767361.511192:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767361.511195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767362.343133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767362.343143:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767362.343144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767362.343151:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343173:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00100000:0.0:1775767362.343181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767362.343182:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343190:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767362.343241:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076734592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767362.343262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767362.343267:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343283:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343287:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343292:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767362.343292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076734592:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767362.343305:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767362.343317:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076734720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767362.343325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343327:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767362.343329:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343331:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343333:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343335:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767362.343345:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076734848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767362.343351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343354:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767362.343355:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343357:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343359:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343362:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767362.343367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767362.343377:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076734976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767362.343388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343391:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767362.343392:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343394:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343398:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343401:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343406:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767362.343410:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350931c00 x1862023076734592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767362.343416:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076735104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767362.343432:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767362.343436:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767362.343440:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076734592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767362.343441:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343446:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343451:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767362.343460:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.343466:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767362.343467:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076735232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767362.343472:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076734592:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767362.343483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767362.343486:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343489:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343496:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343499:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343502:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767362.343510:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076735360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767362.343513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767362.343525:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767362.343529:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767362.343532:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343535:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343537:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343540:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767362.343551:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076735488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767362.343553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767362.343564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767362.343567:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767362.343570:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767362.343574:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350932680 x1862023076734720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767362.343575:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3864 00000100:00080000:1.0:1775767362.343581:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767362.343589:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767362.343595:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767362.343596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076734720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767362.343612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076735616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767362.343616:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.343623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.343627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767362.343633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767362.343658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.343667:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350930700 x1862023076734848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.343676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076734848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.343682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.343684:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.343686:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767362.343689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767362.343703:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.343712:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350932300 x1862023076734976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.343725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076734976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.343740:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.343744:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.343748:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076734976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767362.343754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076735104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767362.343774:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.343785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350930e00 x1862023076735104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.343798:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076735104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.343808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.343813:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.343816:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076735104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767362.343822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767362.343853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.343865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350930000 x1862023076735232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.343878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076735232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.344211:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.344215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.344219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767362.344294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767362.344314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.344322:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350931880 x1862023076735360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.344333:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076735360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.344341:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.344344:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.344348:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767362.344354:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767362.344374:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.344385:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350932a00 x1862023076735488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.344397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076735488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.344403:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.344405:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.344407:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767362.344411:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767362.344425:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767362.344432:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767362/real 1775767362] req@ffffa04350930a80 x1862023076735616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767362.344439:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076735616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767362.344444:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767362.344446:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767362.344448:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076735616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767362.535085:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767362.535101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767363.367112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767363.367133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767363.559146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767363.559229:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767364.391102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767364.391146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767364.583113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767364.583117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767365.415140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767365.415162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767365.607117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767365.607118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767366.439117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767366.439147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767366.631127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767366.631129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767367.463134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767367.463136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463144:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767367.463148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767367.463154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767367.463168:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463177:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463182:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767367.463216:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076735744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767367.463240:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767367.463244:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463249:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463252:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463259:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767367.463272:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076735872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767367.463273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076735744:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767367.463287:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463289:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767367.463290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463294:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463297:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767367.463322:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076736000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767367.463329:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463332:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767367.463333:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463335:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463340:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463343:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463346:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767367.463353:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076736128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767367.463358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767367.463362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463365:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463367:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463370:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463374:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767367.463383:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076735872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767367.463384:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076736256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767367.463397:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767367.463405:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463410:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463424:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767367.463429:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736000:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767367.463430:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463438:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767367.463449:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076736384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767367.463457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736128:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767367.463458:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463465:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767367.463466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463470:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463473:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463477:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463483:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767367.463487:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767367.463500:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076736512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767367.463518:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767367.463522:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463527:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767367.463530:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463536:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00100000:3.0:1775767367.463541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736512:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767367.463542:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463551:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767367.463557:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767367.463578:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076736640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767367.463591:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767367.463594:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767367.463595:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767367.463599:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3869 00000100:00080000:1.0:1775767367.463603:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767367.463608:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767367.463608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736640:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767367.463614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767367.463634:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076736768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767367.463650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767367.655161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767367.655161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767368.487184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767368.487204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767368.679145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767368.679145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767369.511174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767369.511176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767369.703132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767369.703132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775767370.535170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767370.535246:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:1.0:1775767370.535330:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775767370.535372:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767370.535389:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767370.535443:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775767370.535459:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767370.535471:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767370.535478:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350931c00 x1862023076736768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767370.535492:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767370.535511:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076736768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767370.535517:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767370.535532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1775767370.535533:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767370.535543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000800:00000100:1.0:1775767370.535549:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767370.535553:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767370.535563:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767370.535566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350930700 x1862023076736512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076736512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535594:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767370.535607:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350930e00 x1862023076736256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076736256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535633:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767370.535646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350932300 x1862023076736384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535656:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076736384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535670:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767370.535677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350932a00 x1862023076735872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076735872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535697:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535703:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076735872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767370.535710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350931880 x1862023076736000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535721:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076736000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535732:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535734:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767370.535740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350930000 x1862023076736128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076736128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535770:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076736128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767370.535776:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350930a80 x1862023076735744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076735744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535797:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535800:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076735744:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767370.535848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767367/real 1775767370] req@ffffa04350932680 x1862023076736640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767370.535862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076736640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767370.535870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767370.535874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767370.535877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076736640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767370.727104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767370.727104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767371.559171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767371.559198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767371.751115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767371.751121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767372.583136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583144:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767372.583161:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767372.583173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767372.583178:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3874 00010000:00010000:2.0:1775767372.583179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767372.583189:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767372.583196:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767372.583225:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076736896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767372.583253:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583257:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767372.583259:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583264:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583273:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767372.583277:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767372.583289:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767372.583307:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076737024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767372.583307:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076736896:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767372.583325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583329:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767372.583332:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583336:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583340:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767372.583351:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767372.583357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767372.583370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076737152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767372.583382:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767372.583396:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767372.583402:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583407:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583412:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767372.583417:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767372.583424:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767372.583426:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350932680 x1862023076736896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767372.583439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076737280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767372.583450:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076736896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767372.583452:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583461:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767372.583465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583475:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:3.0:1775767372.583476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767372.583479:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767372.583484:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767372.583487:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767372.583494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076736896:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767372.583495:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767372.583513:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076737408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767372.583524:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767372.583528:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767372.583530:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767372.583534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583541:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583553:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767372.583559:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767372.583565:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767372.583577:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767372.583582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350930a80 x1862023076737024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767372.583594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076737536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767372.583625:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583631:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767372.583631:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076737024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767372.583634:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583644:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583651:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767372.583654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767372.583657:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767372.583663:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767372.583664:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767372.583668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767372.583681:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076737664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767372.583681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767372.583697:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583700:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767372.583701:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583706:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:1.0:1775767372.583713:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767372.583717:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767372.583720:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767372.583727:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767372.583737:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350930000 x1862023076737152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767372.583742:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076737792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767372.583768:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076737152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767372.583813:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767372.583816:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767372.583818:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767372.583826:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3874 00000100:00080000:3.0:1775767372.583826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767372.583835:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767372.583836:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767372.583844:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767372.583845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737152:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767372.583854:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767372.583869:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076737920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767372.583958:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767372.583983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.583995:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350931880 x1862023076737280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584007:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076737280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584048:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584052:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767372.584069:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767372.584088:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.584098:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350932a00 x1862023076737408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584109:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076737408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584117:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584120:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584123:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076737408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767372.584128:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767372.584144:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.584152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350932300 x1862023076737536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584162:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076737536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584170:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584176:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767372.584180:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767372.584194:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.584202:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350930e00 x1862023076737664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584212:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076737664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584219:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584225:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767372.584230:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767372.584245:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.584254:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350930700 x1862023076737792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076737792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584281:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767372.584286:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767372.584305:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767372.584316:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767372/real 1775767372] req@ffffa04350931c00 x1862023076737920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767372.584327:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076737920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767372.584335:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767372.584339:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767372.584342:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076737920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767372.775127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767372.776230:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767373.607152:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767373.607263:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767373.799104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767373.799106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767374.631141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767374.631208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767374.823076:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767374.823093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767375.655135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767375.655163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767375.847086:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767375.847110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767376.679113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767376.679148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767376.871098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767376.871122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767377.703138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767377.703171:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767377.703173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767377.703178:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767377.703198:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703207:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703217:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703224:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767377.703274:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076738048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767377.703302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703306:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767377.703308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703317:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703322:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703332:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767377.703346:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076738176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767377.703358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703362:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767377.703364:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703370:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703373:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703379:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767377.703380:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076738048:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767377.703384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767377.703396:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076738304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767377.703431:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767377.703435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703438:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703441:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703453:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703457:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767377.703466:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076738432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767377.703470:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767377.703477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703480:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767377.703482:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703485:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703490:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703492:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703495:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767377.703502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076738560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767377.703510:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767377.703510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349641500 x1862023076738048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767377.703517:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767377.703520:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703529:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703537:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767377.703539:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076738048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767377.703544:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767377.703556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.703561:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767377.703565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076738688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767377.703566:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076738048:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767377.703589:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767377.703592:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703595:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767377.703597:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703600:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703604:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703606:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767377.703610:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703611:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767377.703620:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349642300 x1862023076738176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767377.703623:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076738816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767377.703645:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767377.703647:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076738176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767377.703673:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767377.703676:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703691:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:3.0:1775767377.703695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767377.703697:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767377.703703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767377.703706:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767377.703712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738176:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767377.703715:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767377.703759:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076738944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767377.703775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767377.703778:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767377.703783:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767377.703785:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767377.703789:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3879 00000100:00080000:0.0:1775767377.703793:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767377.703796:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767377.703801:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767377.703822:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767377.703833:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076739072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767377.703837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349643100 x1862023076738304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.703864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076738304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.703871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.703873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.703877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767377.703882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767377.703895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.703901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349640380 x1862023076738432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.703910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076738432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.703919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.703922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.703925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767377.703931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767377.703966:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.703977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349642a00 x1862023076738560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.703989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076738560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.703998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.704001:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.704003:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076738560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767377.704007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767377.704038:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.704045:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349643b80 x1862023076738688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.704052:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076738688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.704063:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.704065:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.704067:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767377.704070:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767377.704101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.704108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349643800 x1862023076738816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.704116:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076738816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.704122:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.704124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.704126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767377.704140:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767377.704167:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.704174:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349641880 x1862023076738944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.704181:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076738944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.704186:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.704187:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.704190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076738944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767377.704193:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767377.704202:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767377.704223:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767377/real 1775767377] req@ffffa04349640a80 x1862023076739072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767377.704230:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076739072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767377.704235:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767377.704237:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767377.704239:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767377.895134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767377.895134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767378.727144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767378.727172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767378.919174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767378.919174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767379.751127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767379.751146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767379.943109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767379.943111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767380.775124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767380.775169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767380.967140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767380.967147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767381.799109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767381.799140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767381.991084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767381.991106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767382.823100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767382.823107:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823113:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767382.823117:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823127:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823137:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775767382.823140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767382.823170:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767382.823218:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076739200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823239:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823242:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767382.823244:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823251:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823255:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823263:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767382.823271:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076739200:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767382.823280:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076739328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823306:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767382.823308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823326:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823331:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823336:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767382.823340:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767382.823350:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076739456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767382.823362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823373:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00100000:3.0:1775767382.823374:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7f480 x1862023076739200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767382.823377:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823392:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823397:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767382.823401:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076739200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767382.823408:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076739584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767382.823409:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823418:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767382.823421:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767382.823424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076739200:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767382.823425:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767382.823429:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823433:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823436:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823438:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823441:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767382.823449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767382.823450:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076739712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767382.823465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823468:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823470:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823472:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823475:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767382.823483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076739840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823493:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823497:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775767382.823497:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767382.823501:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823505:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823509:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823512:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823516:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767382.823520:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7e680 x1862023076739328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767382.823527:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076739968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767382.823546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823549:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767382.823550:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767382.823554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076739328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767382.823555:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3884 00000100:00080000:2.0:1775767382.823559:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823564:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767382.823571:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767382.823576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823582:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767382.823594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076740096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767382.823598:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739456:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767382.823613:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767382.823617:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767382.823620:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767382.823625:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3884 00000800:00000100:3.0:1775767382.823631:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767382.823638:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767382.823643:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823648:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7c380 x1862023076739456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767382.823649:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767382.823663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076740224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767382.823665:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076739456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823688:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823694:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767382.823721:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767382.823738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7dc00 x1862023076739584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823753:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076739584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823763:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767382.823774:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767382.823786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7c700 x1862023076739712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076739712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076739712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767382.823828:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767382.823846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823857:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7c000 x1862023076739840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076739840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823892:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767382.823898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767382.823912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823919:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7f100 x1862023076739968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823926:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076739968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823931:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076739968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767382.823938:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767382.823947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7e300 x1862023076740096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076740096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823968:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.823970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.823972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767382.823975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767382.823984:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767382.823989:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767382/real 1775767382] req@ffffa0434aa7d500 x1862023076740224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767382.823995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076740224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767382.823999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767382.824002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767382.824003:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767383.015106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767383.015114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767383.847196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767383.847200:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767384.039091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767384.039092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767384.871136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767384.871173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767385.063126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767385.063126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767385.895113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767385.895133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767386.087173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767386.087173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767386.919129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767386.919216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767387.111151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767387.111151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767387.943118:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943127:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767387.943131:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767387.943132:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767387.943141:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00100000:1.0:1775767387.943143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767387.943151:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943158:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767387.943190:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076740352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767387.943211:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943213:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767387.943214:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943217:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943220:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943223:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943230:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767387.943242:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076740480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767387.943242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076740352:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767387.943263:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943265:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767387.943267:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943270:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943274:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943277:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943280:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767387.943290:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076740608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767387.943296:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943299:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767387.943302:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767387.943304:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767387.943307:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943310:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943313:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943316:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767387.943324:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076740736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767387.943337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943340:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767387.943342:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943346:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00100000:3.0:1775767387.943349:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7d880 x1862023076740352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767387.943356:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943363:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943368:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767387.943380:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076740352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767387.943383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076740864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767387.943396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767387.943403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767387.943403:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943408:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076740352:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767387.943410:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767387.943414:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943420:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943424:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767387.943429:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767387.943435:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943440:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767387.943451:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767387.943454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076740992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767387.943463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767387.943463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7ea00 x1862023076740480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767387.943471:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767387.943475:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767387.943486:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076740480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767387.943493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943499:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767387.943502:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943508:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767387.943509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076741120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767387.943512:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767387.943523:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767387.943535:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767387.943537:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767387.943540:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767387.943541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943547:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00080000:2.0:1775767387.943552:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767387.943552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7ce00 x1862023076740608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767387.943558:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943567:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767387.943571:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076740608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767387.943579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076741248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767387.943602:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767387.943603:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767387.943607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767387.943609:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767387.943612:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767387.943613:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767387.943621:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3889 00000100:00100000:3.0:1775767387.943624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740736:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767387.943625:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767387.943638:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767387.943642:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767387.943642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767387.943670:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076741376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767387.943680:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7ed80 x1862023076740736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076740736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943697:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943699:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767387.943706:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767387.943719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767387.943726:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7f800 x1862023076740864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076740864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943738:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943742:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076740864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767387.943761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767387.943773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767387.943779:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7fb80 x1862023076740992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076740992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943794:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943796:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076740992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767387.943799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767387.943809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767387.943814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7df80 x1862023076741120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076741120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943828:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767387.943832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767387.943841:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767387.943846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7d880 x1862023076741248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943853:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076741248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767387.943864:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767387.943872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767387.943877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767387/real 1775767387] req@ffffa0434aa7d500 x1862023076741376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767387.943883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076741376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767387.943888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767387.943891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767387.943892:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076741376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767388.135129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767388.135129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767388.967133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767388.967179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767389.159137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767389.159137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767389.991107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767389.991146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767390.183103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767390.183105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767391.015106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767391.015145:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767391.207116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767391.207116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767392.039116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767392.039145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767392.231109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767392.231187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767393.063136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767393.063163:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767393.063163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767393.063168:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767393.063173:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063182:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063193:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767393.063200:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767393.063234:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076741504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767393.063255:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767393.063261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767393.063273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063283:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767393.063296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076741504:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767393.063310:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076741632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767393.063320:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063323:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767393.063325:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063329:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063332:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767393.063335:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063340:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767393.063348:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076741760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767393.063354:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767393.063357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767393.063363:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063366:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063377:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767393.063381:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767393.063390:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350931c00 x1862023076741504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767393.063395:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076741888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767393.063404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767393.063407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076741504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767393.063408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767393.063410:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767393.063416:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767393.063418:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:3.0:1775767393.063420:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767393.063422:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767393.063424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076741504:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767393.063426:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767393.063440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076742016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767393.063444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741632:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767393.063451:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063454:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767393.063455:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063459:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063463:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767393.063465:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767393.063467:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063472:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767393.063477:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350930700 x1862023076741632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767393.063481:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076742144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767393.063490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076741632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767393.063491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063495:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767393.063497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063500:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:3.0:1775767393.063500:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063503:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767393.063504:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767393.063507:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063507:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741632:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767393.063512:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767393.063513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767393.063520:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076742272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767393.063530:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767393.063536:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063539:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767393.063540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350930e00 x1862023076741760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767393.063541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063545:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063549:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767393.063552:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076741760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767393.063553:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063558:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767393.063561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767393.063567:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076742400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767393.063568:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767393.063574:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741888:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767393.063576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767393.063580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767393.063581:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767393.063585:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3894 00000100:00080000:1.0:1775767393.063588:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767393.063591:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767393.063592:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767393.063596:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767393.063600:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350932300 x1862023076741888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767393.063623:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076742528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767393.063637:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076741888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063649:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076741888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767393.063668:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767393.063686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063693:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350932a00 x1862023076742016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767393.063701:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076742016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063708:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063710:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767393.063715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767393.063725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350931880 x1862023076742144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767393.063738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076742144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063743:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063745:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063746:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767393.063749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767393.063758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350930000 x1862023076742272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767393.063776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076742272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767393.063796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767393.063816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350930a80 x1862023076742400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767393.063839:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076742400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063844:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063846:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767393.063854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767393.063873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767393.063883:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767393/real 1775767393] req@ffffa04350932680 x1862023076742528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767393.063896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076742528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767393.063905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767393.063908:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767393.063911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076742528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767393.255103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767393.255108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767394.087122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767394.087142:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767394.279126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767394.279127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767395.111160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767395.111221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767395.303176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767395.303176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767396.135106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767396.135165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767396.327107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767396.327113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767397.159132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767397.159154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767397.351094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767397.351097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767398.183124:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767398.183127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767398.183134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767398.183138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767398.183148:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00100000:0.0:1775767398.183149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767398.183172:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767398.183180:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767398.183219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076742656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767398.183263:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183267:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767398.183268:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767398.183272:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183276:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767398.183280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183289:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767398.183295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076742656:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767398.183303:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076742784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767398.183312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183314:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767398.183316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767398.183319:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183335:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767398.183339:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767398.183351:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767398.183356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076742912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767398.183364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767398.183368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767398.183372:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183374:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767398.183376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183380:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767398.183410:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076743040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767398.183410:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350932680 x1862023076742656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.183442:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076742656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767398.183447:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183455:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767398.183459:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767398.183463:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767398.183464:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183471:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767398.183473:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767398.183477:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767398.183481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076742656:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767398.183484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767398.183495:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076743168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767398.183504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742784:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767398.183519:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183522:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767398.183524:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767398.183529:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767398.183530:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183536:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767398.183540:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767398.183544:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350930a80 x1862023076742784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767398.183545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767398.183594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076743296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767398.183596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076742784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767398.183620:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767398.183621:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767398.183627:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767398.183629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767398.183630:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767398.183638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742784:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767398.183639:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183647:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767398.183651:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742912:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767398.183652:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183661:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767398.183674:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076743424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767398.183682:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767398.183684:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183688:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767398.183690:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767398.183693:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350930000 x1862023076742912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767398.183694:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183705:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767398.183707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076742912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767398.183711:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183723:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767398.183723:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.183729:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.183733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076742912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767398.183739:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767398.183743:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076743552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767398.183756:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767398.183760:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767398.183763:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767398.183768:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767398.183772:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3899 00000100:00080000:1.0:1775767398.183780:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767398.183784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350931880 x1862023076743040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767398.183785:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767398.183793:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767398.183807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076743040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767398.183809:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076743680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767398.183837:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.183842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.183845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767398.183861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767398.183878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767398.183886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350932a00 x1862023076743168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.183907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076743168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767398.183915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.183917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.183919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767398.183924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767398.183964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767398.183977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350932300 x1862023076743296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.183992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076743296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767398.184001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.184004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.184008:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767398.184015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767398.184073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767398.184089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350930e00 x1862023076743424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.184318:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076743424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767398.184329:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.184392:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.184398:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767398.184406:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767398.184443:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767398.184455:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350930700 x1862023076743552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.184467:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076743552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767398.184476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.184479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.184482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767398.184488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767398.184507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767398.184517:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767398/real 1775767398] req@ffffa04350931c00 x1862023076743680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767398.184530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076743680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767398.184540:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767398.184543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767398.184547:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076743680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767398.375121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767398.375121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767399.207149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767399.207193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767399.399105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767399.399105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767400.231104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767400.231133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767400.423136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767400.423136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767401.255114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767401.255140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767401.447091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767401.447108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767402.279098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767402.279130:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767402.471112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767402.471112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767403.303118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767403.303166:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303176:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767403.303182:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303207:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303215:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767403.303221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767403.303286:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076743808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767403.303313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767403.303317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303321:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303325:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303328:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303338:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767403.303350:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076743936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767403.303352:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076743808:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767403.303366:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303369:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767403.303370:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303373:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303375:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303378:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767403.303389:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076744064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767403.303395:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303398:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767403.303399:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303409:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303412:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303415:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303420:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767403.303429:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076744192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767403.303436:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767403.303442:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303445:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767403.303446:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303450:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303453:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303455:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303459:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767403.303467:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076744320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767403.303473:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303476:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767403.303478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00100000:3.0:1775767403.303483:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350931c00 x1862023076743808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767403.303488:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303494:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303498:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767403.303521:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076744448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767403.303534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076743808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767403.303544:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303551:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767403.303554:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767403.303554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767403.303563:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:3.0:1775767403.303564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767403.303572:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767403.303572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076743808:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767403.303581:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303588:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767403.303600:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076744576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767403.303602:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743936:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767403.303616:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767403.303619:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767403.303620:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303624:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00080000:1.0:1775767403.303628:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303632:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767403.303636:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767403.303641:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767403.303646:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076744704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767403.303653:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767403.303655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350930700 x1862023076743936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767403.303658:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767403.303662:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767403.303670:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3904 00000100:00100000:3.0:1775767403.303676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076743936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767403.303679:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767403.303687:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767403.303689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767403.303692:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767403.303695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303699:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076743936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767403.303705:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076744832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767403.303706:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767403.303729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303747:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350930e00 x1862023076744064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.303755:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076744064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.303761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.303763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767403.303769:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767403.303779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350932300 x1862023076744192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.303791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076744192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.303796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.303798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767403.303815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767403.303825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303830:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350932a00 x1862023076744320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.303837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076744320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.303843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.303845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303847:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076744320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767403.303850:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767403.303860:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350931880 x1862023076744448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.303881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076744448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.303887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.303888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303890:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767403.303902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767403.303912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350930000 x1862023076744576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.303926:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076744576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.303957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.303960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.303963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767403.303968:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767403.303985:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.303994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350930a80 x1862023076744704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.304005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076744704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.304012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.304015:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.304018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767403.304023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767403.304036:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767403.304077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767403/real 1775767403] req@ffffa04350932680 x1862023076744832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767403.304088:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076744832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767403.304097:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767403.304100:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767403.304103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076744832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767403.495164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767403.495164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767404.327098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767404.327125:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767404.519149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767404.519157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767405.351114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767405.351156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767405.543118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767405.543118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767406.375131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767406.375189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767406.567135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767406.567135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767407.399114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767407.399164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767407.591128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767407.591157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767408.423118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767408.423123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767408.423134:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767408.423144:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423154:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767408.423157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767408.423163:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767408.423201:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076744960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767408.423228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767408.423233:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767408.423237:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423241:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767408.423245:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076744960:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767408.423276:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767408.423290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076745088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767408.423299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767408.423303:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767408.423306:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423310:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767408.423313:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767408.423317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767408.423319:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767408.423328:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076745216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767408.423336:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423338:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767408.423339:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767408.423342:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423344:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767408.423347:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767408.423351:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767408.423354:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350932680 x1862023076744960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767408.423359:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076745344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767408.423376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423379:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767408.423387:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767408.423391:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076744960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423396:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423406:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767408.423406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423414:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767408.423416:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423421:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076744960:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767408.423425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767408.423441:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076745472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767408.423443:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745088:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767408.423469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767408.423475:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767408.423478:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3910 00000800:00000100:3.0:1775767408.423483:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767408.423484:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767408.423489:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767408.423494:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767408.423498:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350930a80 x1862023076745088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767408.423507:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076745600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767408.423511:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076745088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767408.423532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767408.423536:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767408.423539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767408.423543:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00100000:3.0:1775767408.423544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745088:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767408.423546:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767408.423551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767408.423552:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767408.423560:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767408.423574:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767408.423578:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076745728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767408.423583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350930000 x1862023076745216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767408.423599:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767408.423601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076745216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423604:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00080000:1.0:1775767408.423610:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767408.423612:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767408.423615:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767408.423619:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767408.423622:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767408.423623:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767408.423632:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767408.423636:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076745856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767408.423643:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767408.423646:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767408.423647:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767408.423651:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767408.423654:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3910 00000100:00100000:3.0:1775767408.423660:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350931880 x1862023076745344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423667:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767408.423671:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076745344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767408.423676:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767408.423687:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767408.423689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767408.423705:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076745984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767408.423710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767408.423729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350932a00 x1862023076745472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767408.423752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076745472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767408.423758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423763:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076745472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767408.423767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767408.423778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350932300 x1862023076745600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767408.423792:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076745600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767408.423797:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423799:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423801:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767408.423805:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767408.423815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350930e00 x1862023076745728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767408.423827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076745728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767408.423831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423835:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767408.423838:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767408.423846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423851:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350930700 x1862023076745856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767408.423857:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076745856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767408.423862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767408.423875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767408.423884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767408.423890:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767408/real 1775767408] req@ffffa04350931c00 x1862023076745984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767408.423897:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076745984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767408.423902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767408.423903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767408.423905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076745984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767408.615117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767408.615120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767409.447108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767409.447186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767409.639101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767409.639107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767410.471124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767410.471166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767410.663106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767410.663106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767411.495118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767411.495241:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767411.687105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767411.687150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767412.519139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767412.519174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767412.711112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767412.711124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767413.543153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767413.543153:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543161:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1775767413.543181:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767413.543182:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543204:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543211:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767413.543245:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076746112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543267:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543270:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767413.543271:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543274:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543277:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543288:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767413.543297:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076746240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543304:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767413.543310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767413.543312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076746112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767413.543314:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543319:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543323:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767413.543334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076746368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543351:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543353:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767413.543355:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543372:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543376:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767413.543383:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767413.543387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076746496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543393:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543396:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767413.543397:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543400:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543403:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543405:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767413.543436:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076746624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543442:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543445:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767413.543446:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543449:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543458:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767413.543461:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7e300 x1862023076746112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767413.543464:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543472:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767413.543486:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076746752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767413.543494:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076746112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767413.543495:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543503:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767413.543506:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543513:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543517:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543519:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767413.543519:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767413.543527:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767413.543531:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543538:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076746112:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767413.543539:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076746880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543557:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543559:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767413.543561:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543564:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543567:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767413.543569:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543572:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767413.543578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767413.543581:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076747008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767413.543600:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767413.543602:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767413.543604:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767413.543607:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3915 00000100:00080000:2.0:1775767413.543611:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767413.543616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767413.543618:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767413.543624:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767413.543633:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076747136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767413.543639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7f100 x1862023076746240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.543657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076746240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.543667:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.543671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767413.543681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767413.543702:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.543713:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7c000 x1862023076746368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.543726:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076746368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.543745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.543750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543753:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767413.543760:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767413.543780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.543791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7c700 x1862023076746496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.543804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076746496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.543812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.543816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767413.543825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767413.543845:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.543855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7dc00 x1862023076746624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.543867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076746624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.543876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.543880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543883:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076746624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767413.543888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076746752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767413.543909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.543920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7c380 x1862023076746752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.543932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076746752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.543947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.543950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.543953:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076746752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767413.543959:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076746880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767413.543978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.543989:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7e680 x1862023076746880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.544000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076746880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.544008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.544012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.544015:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076746880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767413.544055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767413.544073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.544084:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7f480 x1862023076747008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.544096:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076747008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.544104:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.544107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.544111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767413.544117:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767413.544134:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767413.544144:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767413/real 1775767413] req@ffffa0434aa7e300 x1862023076747136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767413.544156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076747136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767413.544163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767413.544166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767413.544169:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767413.735110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767413.735110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767414.567129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767414.567163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767414.759116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767414.759116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767415.591103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767415.591160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767415.783096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767415.783100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767416.615112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767416.615144:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767416.807147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767416.807375:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767417.639116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767417.639135:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767417.831143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767417.831143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767418.663122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767418.663149:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663154:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767418.663158:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663165:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663175:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663181:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767418.663205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767418.663217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076747264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767418.663237:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663253:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767418.663255:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663258:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663261:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767418.663273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076747264:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767418.663278:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767418.663297:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076747392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767418.663307:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663310:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767418.663312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663316:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663321:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663324:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767418.663334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076747520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767418.663339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663342:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767418.663343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663346:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663356:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663359:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767418.663389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767418.663393:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076747648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767418.663403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663405:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767418.663407:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663411:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663415:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663419:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663422:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767418.663429:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076747776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767418.663435:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663438:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767418.663440:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663444:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663447:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663453:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767418.663454:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7d500 x1862023076747264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767418.663457:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767418.663483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076747904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767418.663490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076747264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767418.663499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663515:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767418.663522:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663528:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663535:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663545:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767418.663555:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767418.663566:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076748032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767418.663567:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.663583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076747264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767418.663589:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663592:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767418.663594:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663599:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663602:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663605:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663610:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767418.663620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076748160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767418.663626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747392:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767418.663640:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767418.663643:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767418.663645:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767418.663651:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3920 00000100:00080000:2.0:1775767418.663657:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767418.663661:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767418.663667:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767418.663672:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767418.663694:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076748288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767418.663696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7d880 x1862023076747392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.663748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076747392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.663761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.663765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.663769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767418.663776:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767418.663800:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.663812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7df80 x1862023076747520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.663825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076747520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.663834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.663837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.663841:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767418.663848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767418.663870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.663881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7fb80 x1862023076747648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.663894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076747648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.663903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.663906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.663909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767418.663916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767418.663937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.663948:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7f800 x1862023076747776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.663961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076747776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.663970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.663974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.663977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076747776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767418.663984:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767418.664003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.664015:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7ed80 x1862023076747904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.664056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076747904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.664065:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.664069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.664072:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076747904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767418.664080:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767418.664101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.664112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7ce00 x1862023076748032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.664125:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076748032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.664134:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.664137:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.664141:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767418.664164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767418.664185:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.664196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7ea00 x1862023076748160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.664209:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076748160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.664224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.664228:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.664231:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767418.664238:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767418.664259:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767418.664270:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767418/real 1775767418] req@ffffa0434aa7d500 x1862023076748288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767418.664283:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076748288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767418.664292:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767418.664295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767418.664299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076748288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767418.855108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767418.855157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767419.687159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767419.687195:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767419.879151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767419.879151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767420.711106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767420.711128:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767420.903096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767420.903109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767421.735117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767421.735138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767421.927141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767421.927141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767422.759172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767422.759199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767422.951166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767422.951166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767423.783186:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767423.783191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767423.783203:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767423.783208:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767423.783223:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3925 00010000:00010000:3.0:1775767423.783224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767423.783252:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783261:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767423.783310:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076748416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767423.783369:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767423.783374:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783378:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783385:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767423.783412:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076748416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767423.783420:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076748544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783440:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783445:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767423.783466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767423.783473:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783479:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767423.783485:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767423.783490:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783499:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767423.783516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076748672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783528:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767423.783535:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767423.783540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349640700 x1862023076748416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767423.783543:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783551:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783565:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767423.783581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076748416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767423.783591:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076748800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767423.783604:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767423.783611:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767423.783617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767423.783619:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767423.783624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767423.783626:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076748416:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767423.783635:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783640:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783651:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767423.783673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767423.783676:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076748928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783696:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783700:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767423.783703:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767423.783709:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783716:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767423.783716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767423.783728:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783743:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767423.783745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349642680 x1862023076748544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767423.783769:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076749056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767423.783772:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076748544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767423.783809:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767423.783813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767423.783815:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767423.783821:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767423.783826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767423.783834:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00100000:3.0:1775767423.783839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748544:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767423.783841:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783850:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783856:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767423.783857:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767423.783872:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076749184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783880:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783883:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767423.783884:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767423.783895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767423.783896:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.783905:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767423.783910:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767423.783917:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767423.783921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349641180 x1862023076748672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767423.783929:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076749312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767423.783980:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767423.783992:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767423.783994:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767423.784000:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3925 00000100:00080000:0.0:1775767423.784005:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767423.784012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076748672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767423.784016:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767423.784054:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767423.784070:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748800:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767423.784075:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767423.784099:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076749440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767423.784109:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784128:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349641c00 x1862023076748800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784142:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076748800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784156:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767423.784183:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767423.784207:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784220:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349640e00 x1862023076748928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076748928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784249:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784253:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784256:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076748928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767423.784263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767423.784286:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784298:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349640700 x1862023076749056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076749056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784334:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784338:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784341:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767423.784349:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767423.784372:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784384:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349640a80 x1862023076749184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076749184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784410:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784414:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767423.784421:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767423.784442:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784454:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349641880 x1862023076749312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076749312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784474:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784476:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767423.784480:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767423.784495:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767423.784502:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767423/real 1775767423] req@ffffa04349643800 x1862023076749440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767423.784509:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076749440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767423.784516:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767423.784519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767423.784522:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076749440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767423.975193:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767423.975193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767424.807141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767424.807175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767424.999156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767424.999156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767425.831156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767425.831184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767426.023158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767426.023162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767426.855142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767426.855177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767427.047130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767427.047133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767427.879132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767427.879174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767428.071161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767428.071161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767428.903113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767428.903127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767428.903176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767428.903193:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903205:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903220:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903230:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767428.903298:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076749568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767428.903333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767428.903339:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903344:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903349:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903354:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767428.903379:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076749568:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767428.903385:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076749696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767428.903398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767428.903404:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903409:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903413:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903436:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903442:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767428.903456:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076749824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767428.903468:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767428.903471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767428.903479:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903484:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903490:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903502:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767428.903515:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076749952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767428.903526:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350931c00 x1862023076749568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767428.903527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767428.903540:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903549:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903555:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903561:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767428.903561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076749568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767428.903571:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767428.903579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.903587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767428.903590:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076750080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767428.903595:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076749568:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767428.903611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767428.903625:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767428.903629:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749696:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767428.903634:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903648:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903654:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767428.903660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903662:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767428.903676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350930700 x1862023076749696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767428.903679:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076750208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767428.903714:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767428.903719:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767428.903721:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076749696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767428.903724:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903730:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903737:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903744:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767428.903745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.903752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767428.903753:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076750336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767428.903762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749696:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767428.903764:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767428.903770:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767428.903773:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903777:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00100000:3.0:1775767428.903780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749824:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767428.903781:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903787:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903797:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767428.903809:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076750464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767428.903816:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767428.903820:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767428.903821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767428.903824:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767428.903828:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3930 00000100:00080000:1.0:1775767428.903833:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767428.903836:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767428.903839:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767428.903843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350930e00 x1862023076749824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767428.903851:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076750592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767428.903870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076749824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.903880:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.903884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.903888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767428.903896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767428.903918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.903928:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350932300 x1862023076749952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.903941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076749952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.903950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.903953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.903956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076749952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767428.903962:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767428.903977:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.903987:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350932a00 x1862023076750080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.903998:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076750080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.904008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.904011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.904014:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767428.904066:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767428.904090:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.904104:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350931880 x1862023076750208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.904117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076750208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.904127:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.904130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.904134:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767428.904140:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767428.904171:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.904182:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350930000 x1862023076750336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.904194:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076750336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.904204:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.904207:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.904210:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767428.904216:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767428.904231:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.904240:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350930a80 x1862023076750464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.904261:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076750464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.904285:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.904289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.904292:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767428.904298:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767428.904321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767428.904333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767428/real 1775767428] req@ffffa04350932680 x1862023076750592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767428.904346:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076750592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767428.904355:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767428.904358:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767428.904362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076750592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767429.095161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767429.095161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767429.927138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767429.927164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767430.119124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767430.119124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767430.951159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767430.951235:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767431.143124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767431.143124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767431.975180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767431.975217:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767432.167114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767432.167114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767432.999138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767432.999254:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767433.191087:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767433.191088:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767434.023181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767434.023199:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023212:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023240:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023249:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767434.023300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076750720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023325:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023328:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767434.023348:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023355:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023361:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767434.023365:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767434.023369:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023390:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767434.023410:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076750848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775767434.023417:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767434.023417:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076750720:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767434.023444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767434.023450:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023459:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023462:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023465:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767434.023483:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076750976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023492:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767434.023498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023501:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023508:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023512:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023517:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767434.023536:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076751104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767434.023551:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023553:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023557:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023561:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023565:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767434.023574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076751232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023581:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023584:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767434.023586:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023591:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023594:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023598:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767434.023613:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076751360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023622:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767434.023624:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023632:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023636:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023638:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023643:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767434.023651:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076751488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023662:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767434.023664:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023667:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023670:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023673:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023676:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767434.023683:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076751616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767434.023695:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767434.023698:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767434.023699:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767434.023702:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3935 00000100:00080000:1.0:1775767434.023705:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767434.023708:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767434.023713:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767434.023728:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076751744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767434.024512:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767434.024578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767434.024612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076751104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767434.024638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076751232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767434.024672:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767434.024696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767434.024720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767434.024752:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767434.215121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767434.215121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767435.047111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767435.047255:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767435.239088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767435.239092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767436.071214:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767436.071277:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767436.263131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767436.263131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775767437.095274:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775767437.095426:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000100:00100000:0.0:1775767437.095475:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:3.0:1775767437.095501:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775767437.095522:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775767437.095655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350932680 x1862023076750720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767459 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.095725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076750720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767459 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.095746:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.095754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.095759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076750720:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775767437.095855:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095885:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095896:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095906:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095918:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095927:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095937:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767437.095945:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775767437.096007:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350931c00 x1862023076751744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096025:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076751744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096073:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096084:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767437.096094:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350930a80 x1862023076750848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096111:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076750848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096122:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.096139:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350930000 x1862023076750976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096163:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076750976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096172:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096176:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096179:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076750976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.096199:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350931880 x1862023076751104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076751104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096229:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096232:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076751104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767437.096240:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350932a00 x1862023076751232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096253:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076751232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096268:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096272:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076751232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767437.096280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350932300 x1862023076751360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096306:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076751360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096315:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096319:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096322:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.096329:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350930e00 x1862023076751488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096341:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076751488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096350:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096357:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.096364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767434/real 1775767437] req@ffffa04350930700 x1862023076751616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.096376:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076751616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.096384:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.096388:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767437.096391:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076751616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767437.287094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767437.287104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767438.119150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767438.119205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767438.311125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767438.311127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767439.143106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767439.143116:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143122:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767439.143127:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143138:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143149:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143157:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767439.143207:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076751872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767439.143237:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143242:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767439.143244:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143249:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143254:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143260:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143270:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767439.143278:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076751872:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767439.143290:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076752000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767439.143302:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143306:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767439.143309:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143313:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143319:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143323:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143328:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767439.143341:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076752128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767439.143353:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767439.143354:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143359:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767439.143362:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143367:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143372:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143377:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143382:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767439.143395:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076752256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767439.143402:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825ea00 x1862023076751872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767439.143407:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143417:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767439.143421:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143428:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143433:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767439.143438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076751872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767439.143440:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143448:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775767439.143460:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.143468:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767439.143468:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076752384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767439.143476:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076751872:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767439.143489:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143493:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767439.143495:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143500:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143505:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143510:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767439.143512:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752000:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767439.143517:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767439.143532:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076752512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767439.143543:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143547:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767439.143548:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143553:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143558:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767439.143562:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767439.143566:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143572:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767439.143583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825e680 x1862023076752000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.143601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076752000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767439.143604:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076752640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767439.143627:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767439.143628:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143635:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767439.143636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767439.143640:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767439.143645:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752000:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767439.143649:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00080000:3.0:1775767439.143656:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767439.143660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767439.143665:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143671:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767439.143686:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076752768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767439.143696:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767439.143705:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767439.143710:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767439.143712:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767439.143718:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3940 00000100:00100000:0.0:1775767439.143721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825c000 x1862023076752128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767439.143724:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767439.143731:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767439.143736:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767439.143749:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076752896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767439.143780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076752128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.143791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.143795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.143799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767439.143809:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767439.143836:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.143852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825ed80 x1862023076752256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.143867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076752256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.143875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.143879:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.143882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767439.143888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767439.143916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.143927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825e300 x1862023076752384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.143938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076752384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.143962:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.143965:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.143969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076752384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767439.143990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767439.144013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.144024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825f100 x1862023076752512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.144056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076752512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.144065:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.144068:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.144071:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767439.144078:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767439.144103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.144114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825d500 x1862023076752640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.144124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076752640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.144132:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.144135:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.144145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767439.144151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767439.144197:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.144209:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825dc00 x1862023076752768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.144224:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076752768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.144233:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:1.0:1775767439.144233:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767439.144239:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.144243:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767439.144250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767439.144271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767439.144280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767439/real 1775767439] req@ffffa0435825d880 x1862023076752896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767439.144291:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076752896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767439.144298:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767439.144301:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767439.144303:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076752896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767439.335090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767439.335114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767440.167177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767440.167280:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767440.359149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767440.359149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775767441.191130:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767441.191147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767441.383113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767441.383113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767442.215107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767442.215176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767442.407104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767442.407105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767443.239153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767443.239206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767443.431104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767443.431159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767444.263150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767444.263170:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767444.263177:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767444.263182:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767444.263194:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263203:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263210:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767444.263257:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767444.263263:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076753024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767444.263298:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767444.263303:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767444.263305:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767444.263311:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263316:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263321:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767444.263330:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076753024:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767444.263332:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767444.263356:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076753152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767444.263384:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767444.263388:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767444.263390:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767444.263395:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3945 00000800:00000100:3.0:1775767444.263400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767444.263404:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263411:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767444.263418:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767444.263438:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350930700 x1862023076753024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767444.263445:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076753280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767444.263454:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076753024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767444.263466:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767444.263468:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767444.263474:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767444.263475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767444.263480:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767444.263480:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076753024:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767444.263490:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263496:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263501:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767444.263507:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767444.263509:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767444.263528:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076753408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767444.263541:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767444.263542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767444.263548:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767444.263551:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767444.263556:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00100000:3.0:1775767444.263561:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350930e00 x1862023076753152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767444.263570:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263577:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767444.263589:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767444.263597:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076753152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767444.263608:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076753536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767444.263611:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.263620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.263626:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767444.263629:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767444.263633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767444.263636:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767444.263640:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767444.263646:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263653:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767444.263657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767444.263663:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767444.263669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350932300 x1862023076753280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767444.263670:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767444.263691:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076753664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767444.263706:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076753280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767444.263712:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767444.263715:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767444.263718:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767444.263720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767444.263724:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767444.263725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767444.263734:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263741:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767444.263744:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753408:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767444.263761:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767444.263761:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767444.263770:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767444.263772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350932a00 x1862023076753408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767444.263784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076753408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767444.263791:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076753792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767444.263791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.263798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.263807:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767444.263815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753536:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767444.263835:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767444.263839:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767444.263841:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767444.263846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767444.263851:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00080000:1.0:1775767444.263858:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767444.263862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350931880 x1862023076753536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767444.263868:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767444.263877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076753536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767444.263878:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767444.263895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.263900:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767444.263901:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076753920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767444.263903:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076753536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767444.263913:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767444.263925:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767444.263929:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767444.263932:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767444.263937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767444.263941:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3945 00000100:00100000:3.0:1775767444.263946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350930000 x1862023076753664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767444.263947:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767444.263957:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767444.263961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076753664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767444.263965:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767444.263968:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.263971:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.263974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767444.263984:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076754048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767444.263988:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767444.264008:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767444.264017:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350930a80 x1862023076753792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767444.264056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076753792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767444.264066:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.264069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.264073:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767444.264080:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767444.264098:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767444.264106:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350931c00 x1862023076753920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767444.264114:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076753920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767444.264134:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.264136:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.264139:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076753920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767444.264148:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767444.264162:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767444.264169:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767444/real 1775767444] req@ffffa04350932680 x1862023076754048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767444.264178:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076754048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767444.264183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767444.264185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767444.264187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767444.455143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767444.455151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767445.287100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767445.287163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767445.479143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767445.479183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767446.311134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767446.311192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767446.503126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767446.503133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767447.335145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767447.335203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767447.527147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767447.527147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767448.359162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767448.359194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767448.551141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767448.551141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767449.383188:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767449.383201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767449.383202:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1775767449.383218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767449.383236:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383249:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383262:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383271:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767449.383319:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076754176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767449.383353:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.383357:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767449.383359:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383364:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383369:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383374:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767449.383387:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767449.383397:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076754176:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767449.383412:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076754304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767449.383428:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.383433:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767449.383435:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383441:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383446:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383451:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767449.383458:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767449.383463:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767449.383473:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076754432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767449.383499:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.383503:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767449.383505:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767449.383510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350932680 x1862023076754176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767449.383516:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383528:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383535:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767449.383540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076754176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767449.383549:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767449.383553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.383557:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.383561:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076754176:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767449.383569:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076754560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767449.383584:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767449.383586:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767449.383599:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767449.383602:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383609:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383616:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767449.383622:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767449.383627:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767449.383635:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767449.383635:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350931c00 x1862023076754304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767449.383657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076754304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.383665:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767449.383666:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076754688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767449.383674:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.383685:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767449.383707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767449.383709:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.383719:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767449.383722:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383728:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3951 00000800:00000100:3.0:1775767449.383732:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767449.383740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350930a80 x1862023076754432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767449.383742:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383757:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767449.383757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076754432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.383772:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767449.383774:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767449.383778:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.383784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767449.383803:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076754816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767449.383804:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754560:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767449.383833:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767449.383833:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767449.383841:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767449.383844:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767449.383848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350930000 x1862023076754560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767449.383854:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383862:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.383872:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767449.383872:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076754560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767449.383883:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767449.383898:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.383905:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767449.383906:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076754944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767449.383915:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767449.383937:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754688:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767449.383938:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.383955:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767449.383958:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.383964:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.383972:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767449.383976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767449.383983:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767449.383990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350931880 x1862023076754688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767449.383991:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767449.384016:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076755072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767449.384018:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076754688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.384056:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.384061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.384089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076754688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767449.384110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767449.384113:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767449.384119:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767449.384122:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767449.384129:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3951 00000100:00080000:1.0:1775767449.384135:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767449.384141:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767449.384151:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767449.384152:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767449.384166:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350932a00 x1862023076754816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767449.384175:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076755200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767449.384190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076754816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.384209:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.384215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.384219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767449.384231:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767449.384254:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767449.384265:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350932300 x1862023076754944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767449.384277:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076754944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.384301:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.384305:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.384308:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076754944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767449.384323:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767449.384342:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767449.384350:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350930e00 x1862023076755072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767449.384358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076755072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.384364:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.384366:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.384368:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767449.384372:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767449.384383:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767449.384388:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767449/real 1775767449] req@ffffa04350930700 x1862023076755200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767449.384395:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076755200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767449.384400:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767449.384401:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767449.384404:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767449.575180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767449.575186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767450.407178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767450.407197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767450.599135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767450.599135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767451.431139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767451.431170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767451.623131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767451.623131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767452.455135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767452.455160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767452.647115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767452.647115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767453.479126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767453.479174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767453.671119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767453.671119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767454.503170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767454.503174:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503181:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767454.503185:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503194:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503204:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503210:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503228:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767454.503246:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076755328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503269:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503273:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767454.503274:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503279:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503281:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503284:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767454.503292:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767454.503301:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076755456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503310:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767454.503310:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076755328:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767454.503315:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767454.503319:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503323:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503325:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503342:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767454.503346:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767454.503356:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076755584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503363:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503366:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767454.503367:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503369:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503374:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767454.503374:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767454.503380:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767454.503383:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767454.503392:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076755712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503399:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503401:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767454.503402:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503405:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503408:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503413:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503413:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349643b80 x1862023076755328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767454.503419:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767454.503430:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076755840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767454.503436:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076755328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767454.503441:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767454.503446:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767454.503448:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767454.503452:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767454.503452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503457:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076755328:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767454.503460:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503474:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503478:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503482:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755456:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767454.503483:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767454.503498:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076755968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503507:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767454.503511:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767454.503512:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767454.503516:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767454.503521:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503525:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503528:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767454.503531:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767454.503533:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349642a00 x1862023076755456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767454.503542:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076756096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503554:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503557:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767454.503560:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767454.503560:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076755456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767454.503567:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503573:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503577:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767454.503578:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503583:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503589:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755456:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767454.503590:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767454.503596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767454.503602:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076756224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767454.503610:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767454.503612:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767454.503614:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767454.503617:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767454.503620:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3956 00000100:00080000:0.0:1775767454.503625:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767454.503628:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349640380 x1862023076755584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767454.503632:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767454.503645:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076756352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767454.503652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076755584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503662:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767454.503678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767454.503712:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349643100 x1862023076755712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503729:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076755712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767454.503749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767454.503767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503779:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349642300 x1862023076755840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503792:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076755840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503804:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503807:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076755840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767454.503826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767454.503846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349641500 x1862023076755968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076755968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076755968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767454.503880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767454.503892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349643b80 x1862023076756096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503906:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076756096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503912:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503914:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767454.503918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767454.503927:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349643800 x1862023076756224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076756224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767454.503950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767454.503958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767454.503963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767454/real 1775767454] req@ffffa04349641880 x1862023076756352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767454.503969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076756352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767454.503975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767454.503977:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767454.503979:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076756352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767454.695084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767454.695105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767455.527119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767455.527319:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767455.719100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767455.719100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767456.551134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767456.551137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767456.743085:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767456.743094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767457.575114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767457.575170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767457.767101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767457.767102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767458.599104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767458.599140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767458.791137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767458.791137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767459.623158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767459.623158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767459.623170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767459.623176:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775767459.623177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767459.623191:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623206:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623216:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767459.623278:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076756480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767459.623307:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623311:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623314:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623337:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623348:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767459.623363:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076756608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767459.623363:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076756480:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767459.623392:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623395:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767459.623396:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623400:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623403:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623407:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623411:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767459.623421:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076756736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623429:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623431:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767459.623432:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623435:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623437:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623440:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623443:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767459.623452:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076756864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767459.623461:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767459.623469:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767459.623471:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623475:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623480:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623485:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623489:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767459.623498:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076756992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767459.623512:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623517:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00100000:3.0:1775767459.623522:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350930700 x1862023076756480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767459.623523:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623530:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623534:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767459.623550:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076757120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623562:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767459.623564:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767459.623570:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076756480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767459.623575:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623578:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623585:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767459.623605:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767459.623611:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076757248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767459.623616:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.623638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076756480:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767459.623639:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767459.623654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767459.623659:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623663:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623666:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623670:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623675:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767459.623681:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076757376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767459.623689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767459.623690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767459.623697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767459.623700:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767459.623707:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3961 00000100:00080000:1.0:1775767459.623709:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767459.623712:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767459.623716:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767459.623726:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076757504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767459.623738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.623759:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350930e00 x1862023076756608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.623779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076756608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.623788:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.623792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.623795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767459.623802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767459.623824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.623834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350932300 x1862023076756736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.623846:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076756736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.623853:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.623856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.623859:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767459.623866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767459.623883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.623892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350932a00 x1862023076756864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.623903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076756864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.623916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.623919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.623923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767459.623929:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767459.623944:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.623953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350931880 x1862023076756992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.623965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076756992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.623992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.623997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.624000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076756992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767459.624006:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767459.624053:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.624066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350930000 x1862023076757120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.624079:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076757120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.624088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.624091:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.624095:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767459.624102:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767459.624122:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.624132:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350930a80 x1862023076757248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.624144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076757248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.624152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.624155:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.624159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767459.624181:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767459.624203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.624215:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350931c00 x1862023076757376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.624228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076757376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.624237:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.624240:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.624243:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767459.624248:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767459.624266:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767459.624277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767459/real 1775767459] req@ffffa04350932680 x1862023076757504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767459.624289:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076757504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767459.624297:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767459.624300:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767459.624304:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076757504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767459.815112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767459.815155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767460.647110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767460.647187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767460.839128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767460.839128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767461.671120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767461.671122:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767461.863092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767461.863096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767462.695143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767462.695222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767462.887178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767462.887179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767463.719137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767463.719231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767463.911136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767463.911136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767464.743108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767464.743121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767464.743148:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743156:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743165:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743171:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767464.743203:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076757632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767464.743227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767464.743232:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743242:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743247:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743251:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076757632:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767464.743262:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1775767464.743271:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767464.743282:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076757760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767464.743294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767464.743300:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743304:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743309:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767464.743314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767464.743317:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743323:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767464.743334:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076757888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767464.743347:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767464.743352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350932680 x1862023076757632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767464.743352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767464.743355:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743362:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743366:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743373:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743380:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767464.743384:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076757632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743393:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767464.743397:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076758016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767464.743409:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743414:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076757632:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767464.743415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767464.743425:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743430:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743433:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743437:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743441:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076757760:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767464.743442:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767464.743459:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076758144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767464.743467:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767464.743472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767464.743472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767464.743476:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743480:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00100000:0.0:1775767464.743487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350931c00 x1862023076757760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767464.743491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743499:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743504:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767464.743505:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076757760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743515:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767464.743517:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076758272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767464.743522:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743529:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076757760:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767464.743534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767464.743539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076757888:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767464.743540:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767464.743548:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743552:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767464.743555:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743558:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767464.743562:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767464.743569:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076758400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767464.743573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350930a80 x1862023076757888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767464.743591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743594:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767464.743596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076757888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767464.743597:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767464.743606:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767464.743613:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00100000:0.0:1775767464.743616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076757888:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767464.743618:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767464.743623:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758016:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767464.743624:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743627:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767464.743636:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076758528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767464.743654:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767464.743657:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767464.743660:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767464.743661:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767464.743666:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3966 00000100:00080000:1.0:1775767464.743671:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767464.743674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350930000 x1862023076758016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767464.743677:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767464.743684:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767464.743691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076758016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767464.743697:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076758656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767464.743704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743712:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767464.743731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767464.743744:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350931880 x1862023076758144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767464.743757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076758144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767464.743770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767464.743779:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350932a00 x1862023076758272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767464.743801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076758272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767464.743814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767464.743825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743831:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350932300 x1862023076758400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767464.743838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076758400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743846:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767464.743849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767464.743858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350930e00 x1862023076758528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767464.743870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076758528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767464.743891:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767464.743903:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767464.743909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767464/real 1775767464] req@ffffa04350930700 x1862023076758656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767464.743915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076758656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767464.743936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767464.743938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767464.743940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076758656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767464.935116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767464.935119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767465.767102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767465.767150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767465.959115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767465.959124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767466.791128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767466.791211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767466.983143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767466.983144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767467.815127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767467.815140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767468.007089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767468.007101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767468.839129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767468.839191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767469.031121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767469.031122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767469.863123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767469.863135:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863145:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:1.0:1775767469.863153:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863159:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767469.863177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767469.863192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767469.863194:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076758784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767469.863232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863236:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767469.863247:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863257:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00100000:2.0:1775767469.863263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076758784:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767469.863276:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863281:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767469.863290:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767469.863304:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076758912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767469.863315:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767469.863318:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767469.863324:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767469.863326:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863332:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:1.0:1775767469.863337:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863342:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767469.863348:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767469.863351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350930700 x1862023076758784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767469.863365:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076759040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767469.863375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076758784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767469.863391:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767469.863392:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767469.863398:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:2.0:1775767469.863398:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.863406:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076758784:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767469.863407:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863415:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767469.863422:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767469.863428:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767469.863438:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076759168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767469.863456:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767469.863468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767469.863472:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350930e00 x1862023076758912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767469.863480:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863491:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00100000:2.0:1775767469.863491:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076758912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863499:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767469.863502:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767469.863507:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767469.863507:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.863513:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076758912:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767469.863518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767469.863520:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767469.863539:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076759296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767469.863542:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767469.863553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350932300 x1862023076759040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.863562:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076759040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863564:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767469.863575:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863586:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:2.0:1775767469.863588:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767469.863592:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767469.863595:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767469.863602:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767469.863602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759040:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767469.863614:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767469.863624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767469.863632:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076759424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767469.863645:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767469.863647:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863652:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767469.863654:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863660:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00100000:2.0:1775767469.863662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350932a00 x1862023076759168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.863674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076759168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863675:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863684:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767469.863684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767469.863691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767469.863695:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767469.863695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767469.863704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767469.863715:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076759552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767469.863721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767469.863730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350931880 x1862023076759296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863731:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767469.863740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767469.863744:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767469.863748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076759296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767469.863753:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:1.0:1775767469.863761:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767469.863762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767469.863772:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767469.863772:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767469.863779:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767469.863784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076759296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767469.863796:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076759680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767469.863807:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767469.863810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767469.863812:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767469.863816:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767469.863822:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3971 00000100:00080000:1.0:1775767469.863827:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767469.863832:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767469.863837:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767469.863842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767469.863854:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076759808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767469.863859:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350930000 x1862023076759424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.863890:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076759424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767469.863919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767469.863923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.863927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767469.863932:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767469.863947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767469.863956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350930a80 x1862023076759552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.863964:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076759552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767469.863969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767469.863971:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.863973:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767469.863981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767469.863990:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767469.864003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350931c00 x1862023076759680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.864011:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076759680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767469.864016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767469.864018:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.864020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767469.864023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767469.864059:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767469.864071:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767469/real 1775767469] req@ffffa04350932680 x1862023076759808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767469.864080:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076759808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767469.864086:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767469.864089:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767469.864091:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076759808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767470.055123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767470.055125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767470.887129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767470.887249:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767471.079134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767471.079135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767471.911173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767471.911199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767472.103115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767472.103122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767472.935166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767472.935279:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767473.127150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767473.127150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767473.959120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767473.959206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767474.151114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767474.151157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767474.983113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767474.983124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767474.983129:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983140:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983150:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983158:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767474.983167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767474.983199:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076759936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767474.983222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767474.983226:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983229:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983232:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983235:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983243:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767474.983255:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076760064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767474.983261:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076759936:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767474.983271:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767474.983290:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983297:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983303:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983308:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983314:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767474.983315:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767474.983332:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076760192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767474.983340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767474.983344:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983347:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983351:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767474.983352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350932680 x1862023076759936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983356:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983364:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767474.983373:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076759936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767474.983376:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076760320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767474.983395:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767474.983400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767474.983402:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767474.983408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767474.983409:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076759936:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767474.983412:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983420:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983424:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983428:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983431:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760064:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767474.983437:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1775767474.983450:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767474.983458:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076760448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767474.983462:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350931c00 x1862023076760064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983483:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775767474.983483:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076760064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983486:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767474.983494:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983496:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983499:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760064:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767474.983503:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00100000:2.0:1775767474.983507:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760192:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767474.983508:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983515:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983520:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767474.983528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767474.983532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076760576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767474.983538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350930a80 x1862023076760192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767474.983546:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983554:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983559:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983562:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983566:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767474.983567:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076760192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983575:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767474.983576:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076760704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767474.983581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983589:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760192:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767474.983592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767474.983597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760320:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767474.983598:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983609:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983612:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767474.983614:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983618:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767474.983626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767474.983631:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076760832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767474.983638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350930000 x1862023076760320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983640:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767474.983647:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767474.983650:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767474.983657:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3976 00000100:00080000:1.0:1775767474.983660:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767474.983660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076760320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767474.983664:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767474.983671:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767474.983672:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767474.983680:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076760960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767474.983684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767474.983710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767474.983724:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350931880 x1862023076760448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767474.983738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076760448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076760448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767474.983760:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767474.983773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983779:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350932a00 x1862023076760576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767474.983786:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076760576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983794:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767474.983799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767474.983807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350932300 x1862023076760704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767474.983818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076760704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983836:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767474.983849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767474.983863:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350930e00 x1862023076760832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767474.983877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076760832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767474.983890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767474.983898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767474.983903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767474/real 1775767474] req@ffffa04350930700 x1862023076760960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767474.983910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076760960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767474.983915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767474.983916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767474.983918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076760960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767475.175110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767475.175110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767476.007096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767476.007164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767476.199122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767476.199123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767477.031135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767477.031173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767477.223093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767477.223108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767478.055114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767478.055145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767478.247147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767478.247147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767479.079117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767479.079145:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767479.271112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767479.271112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767480.103098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767480.103105:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103110:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767480.103114:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103122:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103132:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103138:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767480.103145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767480.103172:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076761088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767480.103206:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103209:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767480.103210:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103213:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103217:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103220:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767480.103228:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767480.103239:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076761216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767480.103242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076761088:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767480.103256:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767480.103261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103266:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103270:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103274:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767480.103280:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767480.103293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076761344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767480.103302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767480.103311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767480.103318:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103324:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00100000:2.0:1775767480.103336:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350930700 x1862023076761088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767480.103338:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103348:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103355:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076761088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767480.103356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767480.103369:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103373:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103378:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076761088:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767480.103379:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076761472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767480.103393:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767480.103399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767480.103401:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103409:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103415:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103419:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767480.103424:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767480.103427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767480.103434:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350930e00 x1862023076761216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767480.103441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076761600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767480.103449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103453:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767480.103455:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767480.103458:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076761216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767480.103460:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103466:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103473:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767480.103473:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767480.103476:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767480.103480:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767480.103487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076761728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767480.103492:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767480.103500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767480.103511:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103513:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767480.103515:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103519:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3981 00000800:00000100:2.0:1775767480.103523:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767480.103527:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103532:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103536:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350932300 x1862023076761344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767480.103539:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767480.103550:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076761344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103558:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767480.103565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076761856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767480.103573:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761472:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767480.103576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767480.103582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767480.103586:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00080000:1.0:1775767480.103590:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767480.103593:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767480.103597:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767480.103601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767480.103605:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350932a00 x1862023076761472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767480.103613:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076761984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767480.103619:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076761472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767480.103627:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767480.103635:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767480.103636:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767480.103637:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767480.103646:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767480.103647:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3981 00000100:00100000:2.0:1775767480.103650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761472:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767480.103658:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767480.103663:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767480.103668:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767480.103672:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767480.103678:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076762112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767480.103690:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350931880 x1862023076761600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767480.103737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076761600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103754:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076761600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767480.103761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767480.103801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350930000 x1862023076761728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767480.103818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076761728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103824:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767480.103831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767480.103842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350930a80 x1862023076761856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767480.103855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076761856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103865:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767480.103873:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767480.103886:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350931c00 x1862023076761984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767480.103899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076761984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103905:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103907:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076761984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767480.103917:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076762112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767480.103927:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767480.103933:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767480/real 1775767480] req@ffffa04350932680 x1862023076762112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767480.103944:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076762112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767480.103952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767480.103955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767480.103964:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076762112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767480.295101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767480.295107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767481.127143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767481.127177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767481.319089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767481.319113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767482.151123:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767482.151205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767482.343146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767482.343146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767483.175108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767483.175153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767483.367125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767483.367127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767484.199123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767484.199173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767484.391141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767484.391141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767485.223097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767485.223109:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767485.223114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767485.223118:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3986 00010000:00010000:3.0:1775767485.223120:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767485.223129:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223135:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767485.223168:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076762240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767485.223193:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223195:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223198:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223201:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767485.223217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076762368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767485.223233:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076762240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767485.223238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767485.223242:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223247:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223249:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223252:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223256:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767485.223265:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076762496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767485.223277:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223280:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223282:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223284:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767485.223294:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076762624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223300:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767485.223304:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767485.223309:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767485.223311:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223314:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223316:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223325:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767485.223333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076762752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223342:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767485.223343:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223346:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223350:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223354:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223356:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350932680 x1862023076762240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767485.223357:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767485.223366:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076762880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223385:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767485.223386:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223390:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223392:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767485.223394:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076762240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767485.223397:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767485.223416:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767485.223419:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076763008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767485.223425:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223435:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076762240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767485.223438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767485.223454:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767485.223460:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00100000:2.0:1775767485.223465:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767485.223473:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223481:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223487:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767485.223495:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767485.223499:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076763136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767485.223508:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767485.223510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767485.223512:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767485.223516:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350931c00 x1862023076762368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767485.223518:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3986 00000100:00080000:1.0:1775767485.223522:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767485.223528:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767485.223535:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767485.223545:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076763264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767485.223551:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076762368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223571:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223575:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767485.223587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767485.223613:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350930a80 x1862023076762496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.223639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076762496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223651:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223655:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767485.223661:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767485.223681:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223692:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350930000 x1862023076762624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.223703:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076762624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223711:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767485.223748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767485.223769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350931880 x1862023076762752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.223790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076762752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223801:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223805:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076762752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767485.223810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076762880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767485.223827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350932a00 x1862023076762880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.223850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076762880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076762880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767485.223871:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767485.223891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350932300 x1862023076763008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.223926:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076763008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.223935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.223939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.223943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767485.223965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767485.223987:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.223999:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350930e00 x1862023076763136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.224012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076763136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.224021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.224025:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.224053:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767485.224060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767485.224082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767485.224093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767485/real 1775767485] req@ffffa04350930700 x1862023076763264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767485.224107:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076763264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767485.224123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767485.224127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767485.224131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076763264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767485.415121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767485.415122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767486.247111:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767486.247143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767486.439136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767486.439230:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767487.271100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767487.271148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767487.463119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767487.463143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767488.295133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767488.295226:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767488.487113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767488.487130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767489.319150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767489.319209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767489.511101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767489.511102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767490.343142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767490.343149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767490.343154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767490.343163:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343171:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343181:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343187:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767490.343221:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076763392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767490.343244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767490.343248:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343251:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343254:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343257:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767490.343265:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767490.343274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076763520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767490.343279:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076763392:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767490.343296:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767490.343300:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343303:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343306:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343309:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767490.343313:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767490.343321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076763648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767490.343327:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767490.343331:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343333:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343335:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343339:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767490.343343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767490.343345:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767490.343354:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076763776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767490.343360:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343363:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767490.343364:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343367:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343377:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343380:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767490.343384:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767490.343393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076763904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767490.343401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7e300 x1862023076763392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767490.343401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343405:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767490.343407:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343413:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343418:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343422:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767490.343425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767490.343432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076764032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767490.343458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343460:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767490.343462:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343466:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343468:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343472:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076763392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767490.343474:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767490.343487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076764160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767490.343488:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343496:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343500:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076763392:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767490.343513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767490.343515:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767490.343517:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343520:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343523:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343526:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343529:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763520:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767490.343532:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767490.343543:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076764288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767490.343551:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767490.343556:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767490.343556:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767490.343558:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767490.343563:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3992 00000100:00080000:2.0:1775767490.343566:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767490.343569:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343572:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7f480 x1862023076763520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767490.343572:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767490.343581:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076764416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767490.343586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076763520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343595:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343598:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767490.343608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767490.343623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343629:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7e680 x1862023076763648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343636:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076763648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343658:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343660:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767490.343666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767490.343710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343718:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7c380 x1862023076763776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343726:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076763776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343736:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767490.343740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767490.343751:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7dc00 x1862023076763904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076763904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076763904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767490.343782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767490.343795:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7c700 x1862023076764032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343809:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076764032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767490.343821:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767490.343830:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7c000 x1862023076764160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076764160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767490.343878:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767490.343889:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7f100 x1862023076764288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076764288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767490.343921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767490.343932:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767490.343936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767490/real 1775767490] req@ffffa0434aa7e300 x1862023076764416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767490.343943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076764416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767490.343949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767490.343950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767490.343952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076764416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767490.535117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767490.535117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767491.367166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767491.367215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767491.559105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767491.559115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767492.391168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767492.391213:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767492.583153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767492.583153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767493.415120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767493.415162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767493.607097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767493.607097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767494.439152:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767494.439172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767494.631098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767494.631103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767495.463124:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463132:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767495.463136:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463145:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 3997 00010000:00010000:1.0:1775767495.463155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767495.463155:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463163:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767495.463200:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076764544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767495.463223:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463225:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767495.463227:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463230:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463232:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463235:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463244:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767495.463252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076764544:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767495.463257:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076764672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767495.463270:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463274:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767495.463276:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463280:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463285:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463289:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463295:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767495.463321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767495.463327:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076764800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767495.463336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463338:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767495.463339:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463342:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463346:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463349:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463353:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767495.463360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076764928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767495.463364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7d500 x1862023076764544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767495.463379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767495.463385:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463389:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463391:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767495.463392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076764544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767495.463395:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767495.463406:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076765056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767495.463431:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767495.463435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767495.463435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767495.463444:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767495.463448:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463451:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767495.463453:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076764544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767495.463456:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463464:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767495.463474:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076765184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767495.463486:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764672:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767495.463500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463503:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767495.463506:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463512:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3997 00000800:00000100:0.0:1775767495.463515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767495.463521:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463527:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463529:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7ea00 x1862023076764672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767495.463536:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767495.463543:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076764672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463550:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767495.463554:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076765312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767495.463555:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463561:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767495.463572:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767495.463579:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463582:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767495.463585:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767495.463590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767495.463595:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00100000:0.0:1775767495.463599:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7ce00 x1862023076764800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767495.463601:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767495.463610:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463614:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076764800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767495.463615:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767495.463639:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463645:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767495.463650:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076765440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767495.463660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764928:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767495.463675:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767495.463678:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767495.463679:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767495.463683:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 3997 00000100:00080000:2.0:1775767495.463687:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767495.463688:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767495.463694:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767495.463699:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767495.463702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7ed80 x1862023076764928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767495.463711:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076765568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767495.463715:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076764928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076764928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767495.463745:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767495.463758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7f800 x1862023076765056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767495.463773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076765056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463781:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463785:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767495.463807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767495.463827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7fb80 x1862023076765184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767495.463852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076765184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767495.463865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767495.463878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7df80 x1862023076765312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767495.463892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076765312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767495.463904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767495.463913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7d880 x1862023076765440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767495.463925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076765440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767495.463946:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767495.463957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767495.463963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767495/real 1775767495] req@ffffa0434aa7d500 x1862023076765568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767495.463969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076765568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767495.463975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767495.463976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767495.463978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076765568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767495.655091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767495.655106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767496.487135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767496.487164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767496.679107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767496.679112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767497.511137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767497.511165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767497.703150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767497.703150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767498.535124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767498.535159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767498.727125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767498.727127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767499.559150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767499.559188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767499.751131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767499.751134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767500.583150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767500.583176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767500.583188:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583192:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767500.583195:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583203:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583212:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583218:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767500.583275:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076765696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583303:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767500.583305:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583309:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583313:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583318:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767500.583339:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076765824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767500.583339:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076765696:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767500.583349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767500.583362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583366:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583369:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583372:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767500.583385:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076765952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583394:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767500.583399:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583411:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583416:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583421:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767500.583430:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767500.583438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076766080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583452:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767500.583454:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583458:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00100000:2.0:1775767500.583458:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767500.583462:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583465:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583470:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767500.583479:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076766208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767500.583481:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766080:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767500.583489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583504:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767500.583506:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583510:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583513:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767500.583516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766208:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767500.583517:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583522:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767500.583532:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076766336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583550:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583553:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767500.583556:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583561:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583566:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583570:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767500.583572:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766336:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767500.583588:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767500.583600:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076766464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583618:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767500.583621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767500.583622:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767500.583624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583631:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767500.583634:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767500.583649:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076766592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767500.583660:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767500.583663:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767500.583677:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767500.583681:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4002 00000100:00080000:0.0:1775767500.583690:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767500.583691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766592:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767500.583695:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767500.583700:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767500.583711:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076766720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767500.583740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767500.775096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767500.775100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767501.607123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767501.607208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767501.799109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767501.799132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767502.631139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767502.631172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767502.823130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767502.823161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767503.655205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767503.655301:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:3.0:1775767503.655373:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775767503.655416:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775767503.655433:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655485:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655500:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655508:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655514:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655523:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655532:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655540:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767503.655577:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767503.655651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349643b80 x1862023076766720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076766720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655706:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767503.655715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349643800 x1862023076766592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076766592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655736:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767503.655767:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349641880 x1862023076766464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076766464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767503.655827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349642680 x1862023076766336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076766336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655851:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655854:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076766336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767503.655860:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349641180 x1862023076766208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655871:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076766208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655880:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767503.655894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349641c00 x1862023076766080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076766080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767503.655949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349640700 x1862023076765824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076765824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.655973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.655976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767503.655982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349640e00 x1862023076765952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.655992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076765952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.655999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.656002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.656005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076765952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767503.656011:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767500/real 1775767503] req@ffffa04349640a80 x1862023076765696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767503.656055:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076765696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767503.656064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767503.656067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767503.656070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076765696:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767503.847125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767503.847136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767504.679159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767504.679183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767504.871150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767504.871151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767505.703168:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767505.703169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767505.703178:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767505.703183:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703195:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00100000:1.0:1775767505.703197:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767505.703205:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767505.703214:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767505.703250:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076766848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703280:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703284:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767505.703287:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703297:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767505.703303:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767505.703325:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076766848:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767505.703335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076766976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767505.703379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703385:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703397:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767505.703402:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775767505.703408:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767505.703427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076767104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703436:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703439:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767505.703440:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703450:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703456:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767505.703456:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349641500 x1862023076766848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767505.703464:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703476:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767505.703490:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076766848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767505.703492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076767232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703519:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767505.703544:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767505.703545:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767505.703556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767505.703557:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00100000:1.0:1775767505.703564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076766848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767505.703570:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767505.703578:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703584:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767505.703597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767505.703599:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076767360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703615:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703619:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767505.703621:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703626:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703630:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775767505.703636:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767505.703639:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703645:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767505.703657:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349642300 x1862023076766976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767505.703660:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076767488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767505.703690:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076766976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767505.703704:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767505.703711:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767505.703712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767505.703713:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767505.703724:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767505.703725:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703734:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767505.703735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076766976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767505.703746:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767505.703749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767104:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767505.703753:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767505.703767:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076767616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767505.703789:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775767505.703790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767505.703796:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767505.703798:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703803:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00080000:0.0:1775767505.703809:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767505.703810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349643100 x1862023076767104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767505.703817:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703827:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767505.703835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076767104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767505.703843:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076767744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767505.703851:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767505.703862:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767505.703862:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767505.703869:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775767505.703870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767505.703873:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767505.703882:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4007 00000100:00100000:1.0:1775767505.703886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767232:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767505.703888:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767505.703897:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767505.703902:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767505.703916:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076767872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767505.703919:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.703936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349640380 x1862023076767232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.703951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076767232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.703960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.703963:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.703967:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767505.703986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767505.704004:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.704013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349642a00 x1862023076767360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.704153:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076767360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.704165:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.704169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.704172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076767360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767505.704178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767505.704202:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.704214:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349641500 x1862023076767488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.704227:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076767488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.704236:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.704239:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.704243:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767505.704248:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767505.704268:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.704280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349640a80 x1862023076767616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.704293:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076767616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.704314:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.704318:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.704322:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767505.704328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767505.704350:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.704362:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349640e00 x1862023076767744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.704375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076767744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.704382:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.704385:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.704388:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767505.704394:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767505.704415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767505.704427:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767505/real 1775767505] req@ffffa04349640700 x1862023076767872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767505.704438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076767872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767505.704445:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767505.704448:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767505.704458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076767872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767505.895096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767505.895100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767506.727134:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767506.727182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767506.919123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767506.919123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767507.751157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767507.751189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767507.943112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767507.943227:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767508.775121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767508.775154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767508.967103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767508.967105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767509.799120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767509.799158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767509.991119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767509.991126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767510.823115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767510.823122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767510.823134:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823144:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4012 00010000:00010000:3.0:1775767510.823149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767510.823155:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823161:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767510.823193:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076768000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767510.823222:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823224:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00080000:0.0:1775767510.823227:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823230:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823238:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767510.823247:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076768128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823255:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775767510.823259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076768000:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767510.823263:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823268:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00080000:0.0:1775767510.823270:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823276:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767510.823299:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076768256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823308:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767510.823309:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823312:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4012 00000800:00000100:1.0:1775767510.823316:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767510.823319:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823324:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823328:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767510.823336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076768384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767510.823346:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823349:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00100000:1.0:1775767510.823352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349641c00 x1862023076768000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767510.823353:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823360:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767510.823378:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076768000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767510.823379:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076768512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767510.823412:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767510.823416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767510.823419:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767510.823420:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823423:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076768000:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767510.823431:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00080000:0.0:1775767510.823435:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823438:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823442:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767510.823448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767510.823453:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076768640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767510.823461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767510.823464:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823467:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00080000:0.0:1775767510.823471:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823491:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775767510.823495:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823500:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767510.823510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349641180 x1862023076768128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767510.823527:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076768768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767510.823530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076768128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.823556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767510.823558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823562:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767510.823564:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767510.823569:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767510.823572:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00100000:1.0:1775767510.823576:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767510.823580:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767510.823584:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767510.823587:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767510.823594:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767510.823604:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076768896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767510.823610:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823618:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349642680 x1862023076768256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767510.823621:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767510.823628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767510.823631:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767510.823638:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4012 00000100:00100000:1.0:1775767510.823641:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076768256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767510.823644:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767510.823652:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767510.823653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767510.823672:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767510.823674:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823678:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767510.823684:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076769024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767510.823685:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767510.823710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349641880 x1862023076768384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.823729:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076768384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.823735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.823737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767510.823747:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767510.823799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349643800 x1862023076768512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.823824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076768512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.823833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.823836:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076768512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767510.823856:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767510.823871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349643b80 x1862023076768640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.823886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076768640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.823891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.823893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767510.823898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767510.823907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349641c00 x1862023076768768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.823935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076768768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.823943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.823946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.823949:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767510.823956:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767510.823976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.823988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349640700 x1862023076768896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.823996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076768896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.824001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.824004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.824006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076768896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767510.824009:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767510.824022:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767510.824027:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767510/real 1775767510] req@ffffa04349640e00 x1862023076769024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767510.824034:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076769024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767510.824067:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767510.824069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767510.824071:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767511.015098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767511.015114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767511.847132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767511.847198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767512.039119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767512.039119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767512.871118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767512.871199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767513.063105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767513.063115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767513.895109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767513.895215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767514.087123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767514.087132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767514.919150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767514.919178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767515.111132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767515.111132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767515.943128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767515.943128:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943138:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767515.943141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943150:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943159:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775767515.943160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767515.943166:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767515.943199:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076769152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943221:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943236:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767515.943237:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943240:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943244:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943248:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943255:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767515.943267:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076769152:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767515.943277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076769280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943286:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943288:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767515.943290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943292:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943295:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943297:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943301:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767515.943308:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076769408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943315:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767515.943318:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943321:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4017 00000800:00000100:1.0:1775767515.943326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767515.943327:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943331:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943340:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767515.943348:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076769536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943355:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943357:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767515.943358:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943361:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943364:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943367:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767515.943372:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7e300 x1862023076769152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767515.943380:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076769664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943389:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767515.943391:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767515.943394:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767515.943395:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076769152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767515.943402:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943407:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943414:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767515.943420:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767515.943424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076769792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767515.943442:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767515.943449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767515.943451:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076769152:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767515.943458:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767515.943466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943472:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00100000:1.0:1775767515.943476:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767515.943477:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943486:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767515.943503:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767515.943511:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076769920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767515.943521:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767515.943523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7f100 x1862023076769280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767515.943524:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767515.943528:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767515.943537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943541:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767515.943544:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076769280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767515.943547:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767515.943555:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767515.943566:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076770048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767515.943573:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767515.943576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767515.943580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767515.943581:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767515.943584:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767515.943585:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767515.943592:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4017 00000100:00080000:2.0:1775767515.943594:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767515.943596:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769408:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767515.943600:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767515.943619:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076770176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767515.943625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7c000 x1862023076769408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943650:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076769408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943656:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943660:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943663:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767515.943667:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767515.943684:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943695:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7c700 x1862023076769536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076769536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767515.943743:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767515.943758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7dc00 x1862023076769664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076769664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943783:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076769664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767515.943786:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767515.943795:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7c380 x1862023076769792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076769792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943814:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943816:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767515.943829:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767515.943844:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7e680 x1862023076769920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076769920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076769920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767515.943876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767515.943887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7f480 x1862023076770048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076770048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767515.943921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767515.943929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767515.943934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767515/real 1775767515] req@ffffa0434aa7e300 x1862023076770176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767515.943940:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076770176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767515.943949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767515.943951:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767515.943953:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767516.135153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767516.135154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767516.967153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767516.967237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767517.159135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767517.159135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775767517.991159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767517.991160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767518.183100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767518.183100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767519.015130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767519.015140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767519.207141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767519.207142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767520.039156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767520.039187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767520.231120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767520.231145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767521.063205:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767521.063212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767521.063214:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767521.063220:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063229:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00100000:1.0:1775767521.063239:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767521.063239:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.063247:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767521.063287:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076770304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767521.063341:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.063349:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767521.063352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063359:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:2.0:1775767521.063367:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.063395:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.063409:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767521.063422:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076770304:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767521.063435:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076770432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767521.063486:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.063492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767521.063495:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063503:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4022 00000800:00000100:1.0:1775767521.063512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767521.063515:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.063526:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.063537:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767521.063572:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076770560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.063579:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7d500 x1862023076770304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767521.063594:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767521.063599:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076770304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767521.063599:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767521.063603:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063612:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:1.0:1775767521.063617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767521.063618:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767521.063626:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767521.063626:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.063635:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767521.063636:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076770304:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767521.063652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076770688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.063662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767521.063662:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.063673:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767521.063675:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063679:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:2.0:1775767521.063690:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775767521.063694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767521.063703:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767521.063710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7d880 x1862023076770432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767521.063711:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767521.063733:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076770816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.063736:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076770432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767521.063765:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.063770:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767521.063775:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063785:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:1.0:1775767521.063790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767521.063791:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.063799:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.063804:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767521.063809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.063814:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767521.063817:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076770944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.063826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767521.063830:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.063834:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767521.063835:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063839:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:2.0:1775767521.063860:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.063865:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.063871:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767521.063885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767521.063890:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076771072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.063923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7df80 x1862023076770560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767521.063958:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767521.063962:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076770560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767521.063963:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767521.063967:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.063974:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:1.0:1775767521.063979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767521.063981:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767521.063987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767521.063988:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767521.063994:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770560:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767521.063995:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767521.064010:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076771200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.064014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770688:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767521.064029:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767521.064032:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767521.064034:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767521.064038:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4022 00000100:00080000:2.0:1775767521.064042:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767521.064045:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767521.064050:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767521.064063:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7fb80 x1862023076770688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767521.064088:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076771328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767521.064090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076770688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064119:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064123:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064134:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767521.064140:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767521.064158:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064166:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7f800 x1862023076770816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767521.064174:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076770816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064181:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064183:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076770816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767521.064189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767521.064199:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064205:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7ed80 x1862023076770944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767521.064213:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076770944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064219:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064221:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064223:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076770944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767521.064226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767521.064236:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064249:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7ce00 x1862023076771072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767521.064257:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076771072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064265:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064268:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767521.064281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767521.064293:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7ea00 x1862023076771200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767521.064308:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076771200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064313:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064315:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064317:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767521.064321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767521.064332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767521.064342:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767521/real 1775767521] req@ffffa0434aa7df80 x1862023076771328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767521.064355:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076771328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767521.064371:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767521.064388:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767521.064393:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076771328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767521.255155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767521.255236:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767522.087144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767522.087215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767522.279106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767522.279144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767523.111185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767523.111244:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767523.303134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767523.303135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767524.135150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767524.135228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767524.327216:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767524.327216:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767525.159140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767525.159175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767525.351149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767525.351173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767526.183090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767526.183125:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183133:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767526.183138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1775767526.183147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767526.183149:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:0.0:1775767526.183160:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767526.183168:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767526.183205:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076771456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767526.183235:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183240:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767526.183242:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767526.183247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:0.0:1775767526.183252:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767526.183257:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767526.183263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076771456:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767526.183267:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767526.183284:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076771584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767526.183298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183302:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767526.183304:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767526.183309:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4027 00000800:00000100:1.0:1775767526.183322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767526.183345:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767526.183352:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767526.183375:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076771712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767526.183386:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349640a80 x1862023076771456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183387:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183396:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767526.183400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767526.183411:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:0.0:1775767526.183416:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767526.183421:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076771456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183422:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767526.183438:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.183443:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767526.183448:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076771840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767526.183449:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076771456:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767526.183464:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183467:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767526.183470:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767526.183474:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767526.183476:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:0.0:1775767526.183483:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767526.183487:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775767526.183498:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767526.183509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076771968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767526.183510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349641500 x1862023076771584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767526.183579:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076771584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183584:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767526.183589:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767526.183600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767526.183601:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:1.0:1775767526.183609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767526.183610:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767526.183616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771584:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767526.183619:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767526.183624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771712:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767526.183627:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767526.183645:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076772096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767526.183649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767526.183673:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767526.183677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349642a00 x1862023076771712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183678:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767526.183683:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767526.183694:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:0.0:1775767526.183700:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767526.183703:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076771712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183708:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183719:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775767526.183731:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.183737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.183740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767526.183748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767526.183749:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076772224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767526.183763:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183767:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767526.183770:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767526.183776:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4027 00000800:00000100:1.0:1775767526.183778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767526.183789:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767526.183792:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349640380 x1862023076771840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767526.183797:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183809:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767526.183816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076771840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767526.183826:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076772352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767526.183844:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767526.183854:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767526.183857:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767526.183858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767526.183866:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4027 00000100:00080000:1.0:1775767526.183866:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.183873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771840:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767526.183875:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767526.183881:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767526.183887:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767526.183890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767526.183910:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767526.183919:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076772480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767526.183921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349643100 x1862023076771968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767526.183952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076771968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767526.183961:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.183964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.183966:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076771968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767526.183970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767526.183982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767526.183988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349642300 x1862023076772096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767526.183996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076772096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767526.184001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.184002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.184009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767526.184013:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767526.184043:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767526.184051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349640a80 x1862023076772224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767526.184059:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076772224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767526.184064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.184066:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.184068:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767526.184072:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767526.184083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767526.184089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349640e00 x1862023076772352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767526.184096:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076772352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767526.184101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.184103:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.184104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767526.184123:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767526.184142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767526.184161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767526/real 1775767526] req@ffffa04349640700 x1862023076772480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767526.184174:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076772480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767526.184179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767526.184181:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767526.184183:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076772480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767526.375106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767526.375106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767527.207124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767527.207156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767527.399151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767527.399229:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767528.231121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767528.231256:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767528.423167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767528.423167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767529.255131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767529.255210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767529.447161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767529.447161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767530.279155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767530.279216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767530.471113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767530.471120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767531.303094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767531.303122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767531.303134:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303146:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4032 00010000:00010000:3.0:1775767531.303146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767531.303162:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303170:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767531.303223:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076772608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767531.303253:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1775767531.303305:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076772608:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767531.303306:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303316:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303322:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303339:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767531.303353:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076772736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767531.303363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767531.303367:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303370:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303374:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303376:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303380:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767531.303388:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076772864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767531.303394:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767531.303401:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767531.303406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303409:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303411:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303414:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303423:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767531.303432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076772992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767531.303439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767531.303443:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767531.303446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349641c00 x1862023076772608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767531.303449:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303454:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303461:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303465:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767531.303473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076772608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767531.303476:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076773120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767531.303498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767531.303499:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767531.303505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767531.303507:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767531.303508:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767531.303514:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00100000:2.0:1775767531.303517:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076772608:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767531.303518:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303524:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303528:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767531.303535:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076773248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767531.303541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767531.303545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775767531.303547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767531.303552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303557:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303562:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303565:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767531.303569:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303570:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767531.303584:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349643b80 x1862023076772736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767531.303585:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076773376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767531.303602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076772736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767531.303623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767531.303624:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767531.303630:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767531.303633:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767531.303634:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767531.303639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767531.303641:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303647:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767531.303650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767531.303652:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303659:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767531.303671:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076773504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767531.303677:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767531.303686:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767531.303689:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349643800 x1862023076772864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767531.303690:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767531.303692:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767531.303695:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4032 00000100:00080000:0.0:1775767531.303697:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767531.303702:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767531.303707:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767531.303716:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076772864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.303724:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767531.303726:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076773632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767531.303730:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.303738:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767531.303747:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767531.303765:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.303775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349641880 x1862023076772992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.303787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076772992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.303792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.303795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.303797:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076772992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767531.303803:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076773120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767531.303814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.303819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349642680 x1862023076773120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.303825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076773120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.303843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.303847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.303850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076773120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767531.303855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767531.303870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.303877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349641180 x1862023076773248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.303884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076773248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.303890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.303892:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.303893:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767531.303910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767531.303921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.303929:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349641c00 x1862023076773376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.303941:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076773376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.303949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.303953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.303963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767531.303970:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767531.303991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.304000:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349640700 x1862023076773504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.304008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076773504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.304014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.304016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.304018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767531.304022:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767531.304113:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767531.304126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767531/real 1775767531] req@ffffa04349640e00 x1862023076773632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767531.304138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076773632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767531.304146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767531.304149:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767531.304152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076773632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767531.495137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767531.495137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767532.327127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767532.327174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767532.519112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767532.519125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767533.351115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767533.351155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767533.543109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767533.543111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767534.375198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767534.375220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767534.567111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767534.567111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767535.399183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767535.399219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767535.591130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767535.591130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767536.423135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767536.423160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767536.423171:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423181:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423192:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423200:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767536.423208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767536.423267:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076773760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767536.423294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423297:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767536.423299:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423302:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423305:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423310:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767536.423321:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767536.423328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076773760:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767536.423333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076773888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767536.423342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767536.423346:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423370:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423374:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423378:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767536.423382:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767536.423394:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076774016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767536.423405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775767536.423407:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767536.423412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767536.423414:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423420:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423425:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423431:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767536.423438:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767536.423451:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076774144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767536.423456:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7d880 x1862023076773760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767536.423468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767536.423474:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423478:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423482:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767536.423484:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076773760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767536.423487:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767536.423495:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775767536.423499:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423505:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767536.423506:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076774272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767536.423518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076773760:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767536.423526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767536.423531:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423535:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423539:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423542:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076773888:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767536.423547:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767536.423561:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076774400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767536.423570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1775767536.423576:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767536.423577:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767536.423583:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423586:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423589:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7d500 x1862023076773888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767536.423593:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767536.423605:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076774528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767536.423608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076773888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767536.423626:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767536.423629:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767536.423630:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767536.423634:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767536.423637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767536.423641:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00100000:1.0:1775767536.423642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076773888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767536.423648:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774016:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767536.423649:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767536.423655:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767536.423659:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767536.423667:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767536.423672:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076774656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767536.423677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7e300 x1862023076774016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767536.423689:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767536.423694:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767536.423696:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767536.423697:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076774016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767536.423703:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4038 00000100:00080000:2.0:1775767536.423708:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767536.423714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767536.423714:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767536.423723:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767536.423723:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767536.423729:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767536.423734:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076774784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767536.423736:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767536.423760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423779:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7f480 x1862023076774144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.423793:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076774144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.423801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423805:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.423808:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767536.423815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767536.423836:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423847:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7e680 x1862023076774272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.423856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076774272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.423863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423865:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.423867:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076774272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767536.423880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767536.423894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7c380 x1862023076774400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.423908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076774400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.423919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.423924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767536.423928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767536.423940:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7dc00 x1862023076774528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.423954:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076774528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.423959:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423962:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.423963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767536.423967:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767536.423976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.423981:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7c700 x1862023076774656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.423987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076774656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.423992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.423994:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.423999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767536.424004:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767536.424016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767536.424021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767536/real 1775767536] req@ffffa0434aa7c000 x1862023076774784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767536.424053:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076774784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767536.424059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767536.424061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767536.424063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076774784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767536.615168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767536.615168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767537.447146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767537.447210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767537.639085:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767537.639129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767538.471111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767538.471160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767538.663112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767538.663113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767539.495137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767539.495230:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767539.687102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767539.687104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767540.519096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767540.519154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767540.711092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767540.711092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767541.543107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767541.543120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767541.543130:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543137:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543148:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543170:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767541.543216:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076774912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767541.543243:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543246:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543250:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543254:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767541.543262:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767541.543274:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076774912:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767541.543275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076775040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543289:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767541.543293:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543296:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543298:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543301:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767541.543304:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767541.543314:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076775168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543323:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767541.543330:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775767541.543334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767541.543336:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543342:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543346:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767541.543351:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00010000:00010000:0.0:1775767541.543352:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767541.543362:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076775296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543370:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767541.543373:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7f100 x1862023076774912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767541.543374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767541.543378:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543387:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543391:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543394:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076774912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767541.543399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775767541.543412:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767541.543413:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076775424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767541.543420:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076774912:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767541.543428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543435:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767541.543437:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543443:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543448:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543451:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767541.543456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767541.543468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767541.543472:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076775552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543481:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767541.543484:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767541.543485:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543488:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543492:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775767541.543493:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767541.543499:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767541.543503:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767541.543509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7c000 x1862023076775040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767541.543514:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076775680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767541.543530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076775040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767541.543535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767541.543539:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767541.543547:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:1.0:1775767541.543548:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767541.543553:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767541.543556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767541.543560:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543563:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775040:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767541.543565:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767541.543570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767541.543578:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076775808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767541.543585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775767541.543589:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767541.543589:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767541.543593:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767541.543603:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7c700 x1862023076775168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767541.543605:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4043 00000100:00080000:2.0:1775767541.543613:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767541.543618:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543620:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076775168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767541.543624:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767541.543635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767541.543639:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076775936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767541.543643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767541.543664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767541.543691:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543703:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7dc00 x1862023076775296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076775296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543724:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543726:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543728:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767541.543732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767541.543745:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7c380 x1862023076775424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076775424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076775424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767541.543780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767541.543790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7e680 x1862023076775552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076775552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767541.543825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767541.543837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7f480 x1862023076775680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076775680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767541.543861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767541.543870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543876:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7e300 x1862023076775808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076775808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767541.543899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767541.543910:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767541.543916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767541/real 1775767541] req@ffffa0434aa7d500 x1862023076775936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767541.543923:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076775936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767541.543928:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767541.543929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767541.543931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076775936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767541.735097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767541.735098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767542.567120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767542.567229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767542.759124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767542.759129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767543.591112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767543.591135:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767543.783103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767543.783104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767544.615174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767544.615205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767544.807118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767544.807118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767545.639176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767545.639246:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767545.831099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767545.831113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767546.663149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767546.663181:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663187:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767546.663190:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767546.663200:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4048 00010000:00010000:0.0:1775767546.663204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767546.663210:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767546.663217:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767546.663254:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076776064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767546.663283:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663304:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767546.663307:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767546.663313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767546.663323:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076776064:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767546.663334:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767546.663352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076776192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767546.663364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663368:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767546.663370:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767546.663374:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663379:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767546.663383:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767546.663388:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767546.663388:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767546.663403:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076776320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767546.663412:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663415:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767546.663417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767546.663421:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663425:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767546.663429:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663437:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350930700 x1862023076776064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767546.663442:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767546.663457:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076776064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767546.663465:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.663468:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767546.663473:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076776448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767546.663475:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076776064:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767546.663491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767546.663496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767546.663498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776192:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767546.663504:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663511:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767546.663516:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767546.663520:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767546.663523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767546.663539:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076776576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767546.663542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350930e00 x1862023076776192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663571:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767546.663574:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076776192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663578:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767546.663583:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767546.663590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767546.663591:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:2.0:1775767546.663597:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767546.663599:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767546.663600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776192:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767546.663606:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776320:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767546.663614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767546.663629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076776704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767546.663632:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767546.663643:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767546.663643:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350932300 x1862023076776320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663652:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775767546.663658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076776320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663662:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767546.663671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767546.663672:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:2.0:1775767546.663677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767546.663680:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767546.663680:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767546.663689:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776448:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767546.663699:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767546.663709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767546.663717:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076776832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767546.663719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350932a00 x1862023076776448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767546.663740:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076776448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663747:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663759:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767546.663763:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767546.663767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767546.663770:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663775:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767546.663776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767546.663783:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776448:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767546.663791:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767546.663806:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076776960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767546.663810:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776576:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767546.663828:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767546.663832:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767546.663834:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767546.663839:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350931880 x1862023076776576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663852:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4048 00000100:00080000:1.0:1775767546.663861:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767546.663865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076776576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767546.663869:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767546.663879:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767546.663879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.663883:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767546.663885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076776576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767546.663893:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767546.663901:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076777088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767546.663909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350930000 x1862023076776704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767546.663928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076776704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767546.663934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.663936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767546.663944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767546.663948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767546.663960:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767546.663967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350930a80 x1862023076776832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767546.663974:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076776832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767546.663979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.663981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767546.663983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767546.663992:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767546.664001:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767546.664006:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350931c00 x1862023076776960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767546.664013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076776960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767546.664018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.664020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767546.664021:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076776960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767546.664024:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767546.664061:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767546.664079:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767546/real 1775767546] req@ffffa04350932680 x1862023076777088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767546.664092:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076777088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767546.664101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767546.664105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767546.664108:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767546.855112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767546.855112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767547.687101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767547.687163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767547.879103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767547.879103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767548.711135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767548.711307:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767548.903160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767548.903160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767549.735118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767549.735216:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767549.927128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767549.927128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767550.759124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767550.759159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767550.951135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767550.951135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767551.783130:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783139:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767551.783144:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767551.783147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767551.783153:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783164:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767551.783170:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767551.783206:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076777216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775767551.783225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767551.783234:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783238:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767551.783240:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767551.783245:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783255:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767551.783260:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767551.783270:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767551.783273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076777216:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767551.783286:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076777344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767551.783299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783314:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767551.783316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767551.783321:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783326:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767551.783328:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767551.783333:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767551.783339:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767551.783353:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076777472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767551.783367:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350932680 x1862023076777216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783379:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767551.783381:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767551.783386:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076777216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783387:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783394:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767551.783402:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767551.783408:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767551.783410:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767551.783418:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076777216:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767551.783426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076777600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767551.783440:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783443:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767551.783445:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767551.783450:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00100000:2.0:1775767551.783454:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767551.783455:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767551.783462:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767551.783474:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1775767551.783478:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767551.783490:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076777728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767551.783501:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350931c00 x1862023076777344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783511:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767551.783515:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767551.783522:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00100000:2.0:1775767551.783523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076777344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783530:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767551.783533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767551.783538:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767551.783540:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777344:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767551.783548:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767551.783553:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767551.783563:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076777856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767551.783571:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767551.783577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775767551.783582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350930a80 x1862023076777472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783583:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767551.783592:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00100000:2.0:1775767551.783597:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076777472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783598:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767551.783607:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767551.783613:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767551.783616:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767551.783618:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767551.783627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767551.783631:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076777984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767551.783642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767551.783643:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767551.783647:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767551.783650:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350930000 x1862023076777600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783658:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767551.783662:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076777600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783666:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783673:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767551.783678:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767551.783679:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767551.783683:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783687:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777600:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767551.783690:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767551.783701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767551.783707:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076778112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767551.783719:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767551.783721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767551.783726:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767551.783728:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767551.783732:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350931880 x1862023076777728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783737:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4053 00000100:00080000:1.0:1775767551.783745:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767551.783747:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076777728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767551.783754:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767551.783758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767551.783762:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767551.783763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076777728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767551.783773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767551.783779:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076778240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767551.783789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767551.783800:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350932a00 x1862023076777856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767551.783813:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076777856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767551.783821:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767551.783824:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767551.783840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767551.783861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767551.783869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350932300 x1862023076777984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767551.783877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076777984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767551.783882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767551.783884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076777984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767551.783896:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076778112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767551.783906:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767551.783912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350930e00 x1862023076778112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767551.783919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076778112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767551.783924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767551.783925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076778112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767551.783931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076778240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767551.783939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767551.783952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767551/real 1775767551] req@ffffa04350930700 x1862023076778240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767551.783960:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076778240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767551.783965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767551.783967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767551.783968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076778240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767551.975121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767551.975121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767552.807125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767552.807180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767552.999154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767552.999161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767553.831109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767553.831186:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767554.023102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767554.023102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767554.855130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767554.855198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767555.047125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767555.047125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767555.879121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767555.879167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767556.071127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767556.071127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767556.903128:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903136:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767556.903141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903149:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00100000:0.0:1775767556.903163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767556.903173:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767556.903179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767556.903180:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767556.903217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076778368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903245:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903249:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767556.903251:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903256:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903261:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903275:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767556.903280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076778368:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767556.903288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076778496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903301:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767556.903302:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903305:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903308:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903310:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903313:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767556.903322:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076778624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903329:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903336:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767556.903338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903343:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4058 00000800:00000100:2.0:1775767556.903344:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767556.903347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903351:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767556.903363:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076778752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903371:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903374:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767556.903375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903377:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903379:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903382:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903385:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767556.903390:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350930700 x1862023076778368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767556.903394:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076778880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903413:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903416:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767556.903417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767556.903420:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076778368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767556.903423:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903428:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903434:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767556.903437:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903443:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767556.903445:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767556.903446:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076778368:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767556.903459:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076779008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767556.903466:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903468:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775767556.903471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778496:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767556.903472:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903481:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767556.903491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767556.903506:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767556.903513:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076779136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767556.903518:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350930e00 x1862023076778496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767556.903540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767556.903542:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076778496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767556.903549:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767556.903552:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903560:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:2.0:1775767556.903561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767556.903567:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767556.903568:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903573:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778496:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767556.903574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903583:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767556.903584:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767556.903597:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076779264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767556.903604:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767556.903607:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767556.903610:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767556.903614:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350932300 x1862023076778624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767556.903618:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767556.903627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4058 00000100:00080000:1.0:1775767556.903631:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767556.903632:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076778624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767556.903637:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767556.903646:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767556.903647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903652:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767556.903657:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076779392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767556.903662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767556.903669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767556.903684:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350932a00 x1862023076778752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076778752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903715:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767556.903721:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767556.903733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350931880 x1862023076778880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903746:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076778880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903753:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903755:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076778880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767556.903758:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767556.903768:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350930000 x1862023076779008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076779008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767556.903811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767556.903824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903830:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350930a80 x1862023076779136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076779136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903843:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767556.903848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767556.903857:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350931c00 x1862023076779264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076779264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767556.903886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767556.903898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767556.903904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767556/real 1775767556] req@ffffa04350932680 x1862023076779392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767556.903911:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076779392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767556.903916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767556.903918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767556.903920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076779392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767557.095103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767557.095123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767557.927118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767557.927139:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767558.119128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767558.119128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767558.951121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767558.951185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767559.143099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767559.143100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767559.975110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767559.975144:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767560.167129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767560.167129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767560.999096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767560.999159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767561.191084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767561.191110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767562.023129:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767562.023132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767562.023141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767562.023145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023155:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023165:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023174:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767562.023196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767562.023217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076779520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023244:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023247:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767562.023248:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023251:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023254:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023258:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023266:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767562.023279:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076779520:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767562.023293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076779648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023305:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767562.023306:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023309:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023312:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023315:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023319:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767562.023326:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076779776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023333:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023336:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767562.023337:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023339:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023341:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023345:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767562.023347:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023349:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767562.023357:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076779904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023366:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767562.023367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023375:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023378:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023382:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767562.023394:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350932680 x1862023076779520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767562.023397:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076780032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023407:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767562.023408:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023411:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023416:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023421:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767562.023434:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076780160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767562.023440:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076779520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767562.023442:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023446:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767562.023449:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023457:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023460:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023461:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023464:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767562.023471:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076780288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767562.023480:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767562.023484:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023488:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767562.023489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767562.023490:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767562.023495:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00100000:2.0:1775767562.023499:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076779520:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767562.023500:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023506:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023511:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767562.023518:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076780416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767562.023523:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767562.023526:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767562.023528:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767562.023532:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767562.023533:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4063 00000100:00080000:1.0:1775767562.023538:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767562.023542:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767562.023545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767562.023553:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076780544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767562.023570:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350931c00 x1862023076779648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023599:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076779648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023609:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023613:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767562.023623:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767562.023643:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350930a80 x1862023076779776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076779776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023682:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767562.023694:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767562.023738:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350930000 x1862023076779904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023758:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076779904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076779904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767562.023772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767562.023782:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023788:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350931880 x1862023076780032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076780032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023800:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767562.023807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767562.023817:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350932a00 x1862023076780160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076780160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023843:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767562.023848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767562.023859:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350932300 x1862023076780288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023873:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076780288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023878:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023882:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767562.023885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767562.023894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350930e00 x1862023076780416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023906:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076780416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.023927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.023929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.023932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767562.023948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767562.023965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767562.023973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767562/real 1775767562] req@ffffa04350930700 x1862023076780544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767562.023999:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076780544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767562.024005:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767562.024008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767562.024010:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076780544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767562.215133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767562.215133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767563.047179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767563.047401:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767563.239131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767563.239139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767564.071165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767564.071246:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767564.263144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767564.263144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767565.095120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767565.095289:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767565.287101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767565.287102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767566.119130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767566.119193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767566.311103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767566.311112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767567.143109:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143117:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767567.143121:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775767567.143125:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767567.143130:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00100000:3.0:1775767567.143139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767567.143160:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143167:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767567.143196:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076780672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143217:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143220:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767567.143221:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143224:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143226:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143229:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143236:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767567.143246:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076780800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143254:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767567.143260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076780672:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767567.143263:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767567.143264:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143269:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143274:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143277:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767567.143292:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076780928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143310:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767567.143312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143315:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143319:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143322:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143326:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767567.143335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076781056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143341:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143344:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767567.143345:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143348:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143350:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143353:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767567.143365:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767567.143366:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076781184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143381:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767567.143387:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143392:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143398:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767567.143401:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780928:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767567.143407:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143421:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767567.143424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767567.143440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076781312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767567.143449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781184:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767567.143451:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143457:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767567.143458:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767567.143462:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143466:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143470:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143475:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767567.143477:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767567.143487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076781440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143495:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143499:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767567.143500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767567.143504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781440:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767567.143507:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143512:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143514:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767567.143528:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076781568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767567.143536:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767567.143539:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767567.143552:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767567.143556:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781568:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767567.143559:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4068 00000100:00080000:1.0:1775767567.143565:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767567.143568:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767567.143573:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767567.143584:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076781696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767567.143616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767567.335114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767567.335115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767568.167118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767568.167201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767568.359100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767568.359101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767569.191120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767569.191162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767569.383099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767569.383110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767570.215105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767570.215211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775767570.215505:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775767570.215558:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767570.215575:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215623:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215633:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215639:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215644:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215649:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215654:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215659:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767570.215664:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767570.215703:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350932680 x1862023076781696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.215737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076781696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.215755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.215764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.215778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767570.215796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350931c00 x1862023076781568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.215844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076781568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.215853:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.215859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.215864:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767570.215876:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350930a80 x1862023076781440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.215914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076781440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.215927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.215932:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.215935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767570.215948:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350930000 x1862023076781312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.215961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076781312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.215969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.215972:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.215976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076781312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767570.215982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350931880 x1862023076781184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.216001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076781184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.216016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.216019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.216022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767570.216059:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350930e00 x1862023076780800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.216073:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076780800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.216083:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.216086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.216089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767570.216096:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350932300 x1862023076780928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.216109:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076780928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.216117:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.216121:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.216124:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076780928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767570.216131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350932a00 x1862023076781056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.216153:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076781056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.216162:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.216167:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.216172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767570.216182:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767567/real 1775767570] req@ffffa04350930700 x1862023076780672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767570.216198:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076780672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.216206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767570.216210:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767570.216213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076780672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767570.407155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767570.407156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767571.239169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767571.239229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767571.431105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767571.431105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767572.263161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767572.263163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767572.263171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767572.263175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767572.263180:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263192:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263201:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263214:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767572.263244:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076781824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767572.263269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767572.263275:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263280:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263284:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263288:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767572.263314:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076781952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767572.263326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767572.263331:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263334:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263338:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263341:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263360:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767572.263393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076782080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767572.263393:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076781824:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767572.263414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263418:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767572.263419:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263424:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263430:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263434:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767572.263453:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076782208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767572.263463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767572.263469:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263472:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263476:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263480:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767572.263486:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263489:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767572.263504:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076782336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767572.263513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767572.263525:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263530:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00100000:2.0:1775767572.263538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349640a80 x1862023076781824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767572.263543:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263552:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263561:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767572.263568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076781824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767572.263577:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076782464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767572.263583:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767572.263589:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263597:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767572.263599:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767572.263602:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767572.263607:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00100000:2.0:1775767572.263612:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076781824:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767572.263613:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263622:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767572.263627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781952:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767572.263628:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767572.263646:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076782592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767572.263656:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263659:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1775767572.263665:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767572.263670:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263676:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263681:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767572.263686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349642300 x1862023076781952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767572.263690:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263699:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767572.263707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076781952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.263719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767572.263723:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076782720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767572.263726:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.263733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076781952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767572.263746:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767572.263750:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767572.263751:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767572.263756:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4073 00000100:00080000:0.0:1775767572.263761:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767572.263766:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767572.263772:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767572.263773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767572.263790:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076782848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767572.263805:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.263820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349643100 x1862023076782080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.263833:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076782080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.263841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.263845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.263848:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767572.263854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767572.263902:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.263913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349640380 x1862023076782208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.263927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076782208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.263936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.263939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.263943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767572.263949:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767572.263978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.263992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349642a00 x1862023076782336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.264003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076782336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.264011:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.264015:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.264018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076782336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767572.264024:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767572.264096:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.264108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349641500 x1862023076782464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.264138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076782464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.264147:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.264150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.264153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767572.264159:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767572.264182:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.264194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349640e00 x1862023076782592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.264206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076782592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.264214:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.264226:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.264229:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767572.264236:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767572.264256:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.264267:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349640700 x1862023076782720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.264280:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076782720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.264301:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.264306:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.264309:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767572.264334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767572.264358:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767572.264373:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767572/real 1775767572] req@ffffa04349641c00 x1862023076782848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767572.264390:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076782848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767572.264398:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767572.264401:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767572.264404:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076782848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767572.455141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767572.455157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767573.287118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767573.287223:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767573.479095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767573.479105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767574.311142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767574.311177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767574.503161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767574.503162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767575.335130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767575.335205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767575.527098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767575.527103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767576.359143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767576.359174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767576.551100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767576.551101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767577.383107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767577.383165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767577.383177:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767577.383182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767577.383187:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383197:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383212:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767577.383245:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076782976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383284:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767577.383286:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383291:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383297:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383303:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383313:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767577.383320:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076782976:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767577.383332:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076783104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383346:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383351:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767577.383353:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383359:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383372:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383377:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383383:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767577.383387:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767577.383402:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076783232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383415:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383419:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767577.383427:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383432:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383437:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383441:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767577.383461:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076783360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383471:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383475:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767577.383478:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767577.383479:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7d880 x1862023076782976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767577.383485:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383492:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383499:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767577.383514:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076782976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767577.383520:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076783488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767577.383538:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767577.383541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383544:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767577.383546:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383550:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383553:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383557:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767577.383578:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076783616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383588:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383591:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767577.383592:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383596:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383601:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383605:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383611:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767577.383623:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076783744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767577.383631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767577.383636:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383640:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383644:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383647:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383652:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767577.383662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767577.383669:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076783872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767577.383670:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076782976:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767577.383684:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767577.383688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767577.383690:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767577.383700:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4079 00000100:00080000:2.0:1775767577.383705:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767577.383709:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767577.383716:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767577.383734:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076784000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767577.383747:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767577.383775:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.383790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7df80 x1862023076783104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.383803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076783104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.383810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.383812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.383815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767577.383821:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767577.383854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.383866:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7ea00 x1862023076783232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.383879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076783232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.383888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.383892:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.383895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767577.383899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767577.383914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.383922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7ce00 x1862023076783360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.383930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076783360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.383935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.383937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.383939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767577.383943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767577.383952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.383958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7ed80 x1862023076783488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.383966:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076783488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.383971:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.383973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.383975:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076783488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767577.383978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767577.383988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.383998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7f800 x1862023076783616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.384010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076783616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.384019:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.384022:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.384046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767577.384055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767577.384073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.384082:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7fb80 x1862023076783744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.384090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076783744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.384095:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.384097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.384099:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767577.384103:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767577.384112:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.384118:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7d500 x1862023076783872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.384125:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076783872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.384130:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.384131:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.384133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076783872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767577.384147:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767577.384168:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767577.384178:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767577/real 1775767577] req@ffffa0434aa7e300 x1862023076784000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767577.384190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076784000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767577.384199:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767577.384203:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767577.384207:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767577.575136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767577.575136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767578.407125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767578.407165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767578.599102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767578.599111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767579.431110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767579.431156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767579.623137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767579.623161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767580.455134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767580.455192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767580.647102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767580.647152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767581.479082:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767581.479125:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767581.671133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767581.671133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767582.503113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767582.503113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767582.503124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767582.503128:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503139:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4084 00010000:00010000:3.0:1775767582.503147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767582.503150:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767582.503158:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767582.503188:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076784128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767582.503210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767582.503216:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503220:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503244:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767582.503245:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076784128:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767582.503254:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767582.503264:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767582.503280:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076784256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767582.503292:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503295:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767582.503297:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767582.503302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767582.503306:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503312:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767582.503317:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767582.503322:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767582.503337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349641180 x1862023076784128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767582.503339:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076784384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767582.503354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767582.503363:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503372:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503376:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767582.503379:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076784128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767582.503384:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767582.503394:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767582.503394:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503401:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503404:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076784128:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767582.503420:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076784512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767582.503423:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767582.503446:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767582.503448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767582.503453:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767582.503457:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349642680 x1862023076784256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767582.503460:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503467:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767582.503472:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503475:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076784256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767582.503478:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767582.503487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767582.503495:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076784640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767582.503495:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767582.503506:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767582.503514:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767582.503518:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503522:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503525:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767582.503528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767582.503531:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767582.503536:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767582.503540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349641880 x1862023076784384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767582.503547:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076784768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767582.503555:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076784384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767582.503556:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767582.503563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767582.503565:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503573:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:3.0:1775767582.503577:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767582.503583:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767582.503585:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767582.503589:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503593:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784384:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767582.503595:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767582.503608:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076784896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767582.503617:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767582.503619:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784512:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767582.503620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767582.503624:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503631:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00080000:0.0:1775767582.503633:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767582.503636:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767582.503639:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767582.503647:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076785024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767582.503653:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767582.503662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767582.503667:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767582.503669:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767582.503675:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4084 00000100:00100000:3.0:1775767582.503677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349643800 x1862023076784512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767582.503682:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767582.503691:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076784512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767582.503701:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767582.503702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503711:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767582.503716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767582.503721:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076785152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767582.503735:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503747:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349643b80 x1862023076784640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767582.503755:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076784640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767582.503762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076784640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767582.503770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767582.503780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349641180 x1862023076784768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767582.503820:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076784768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767582.503825:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767582.503841:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767582.503855:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503866:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349641c00 x1862023076784896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767582.503878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076784896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767582.503887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503890:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503893:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076784896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767582.503899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767582.503918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349640700 x1862023076785024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767582.503951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076785024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767582.503959:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.503961:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.503963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767582.503967:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767582.503982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767582.503988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767582/real 1775767582] req@ffffa04349640e00 x1862023076785152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767582.503996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076785152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767582.504001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767582.504003:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767582.504005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767582.695109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767582.695109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767583.527173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767583.527205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767583.719124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767583.719131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767584.551129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767584.551146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767584.743143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767584.743143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767585.575127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767585.575232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767585.767144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767585.767159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767586.599109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767586.599132:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767586.791114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767586.791139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767587.623124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767587.623130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767587.623132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767587.623137:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623146:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623158:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623166:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767587.623198:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076785280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767587.623218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623221:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00010000:00010000:2.0:1775767587.623221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767587.623224:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623229:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623232:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623235:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767587.623243:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767587.623257:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076785408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767587.623259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076785280:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767587.623286:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623290:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767587.623303:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623310:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623315:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623333:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767587.623340:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767587.623353:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076785536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767587.623359:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767587.623370:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623373:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767587.623375:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623378:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623383:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623385:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767587.623389:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767587.623396:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076785664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767587.623402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767587.623405:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623416:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623419:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623422:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767587.623426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767587.623441:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076785792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767587.623442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349641500 x1862023076785280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767587.623478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623485:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767587.623488:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623494:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00100000:3.0:1775767587.623495:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076785280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767587.623501:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623515:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767587.623519:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767587.623520:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767587.623527:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623532:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076785280:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767587.623539:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076785920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767587.623548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767587.623552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767587.623554:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623556:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623559:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767587.623562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767587.623566:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767587.623575:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076786048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767587.623585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767587.623590:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767587.623600:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767587.623604:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623611:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623614:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623622:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349642a00 x1862023076785408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767587.623628:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767587.623652:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076786176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767587.623663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076785408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767587.623675:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767587.623685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767587.623685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767587.623693:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767587.623695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623702:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785408:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767587.623706:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4089 00000100:00080000:0.0:1775767587.623719:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767587.623726:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623727:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785536:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767587.623750:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767587.623763:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076786304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767587.623784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349640380 x1862023076785536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.623810:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076785536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.623817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.623819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767587.623827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767587.623842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349643100 x1862023076785664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.623855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076785664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.623866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.623868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767587.623874:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767587.623884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349642300 x1862023076785792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.623906:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076785792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.623912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.623915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076785792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767587.623921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767587.623933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623940:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349640a80 x1862023076785920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.623947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076785920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.623953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.623955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.623957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076785920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767587.623976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767587.623988:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.623994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349641500 x1862023076786048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.624001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076786048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.624006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.624008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.624010:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767587.624014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767587.624024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.624056:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349640e00 x1862023076786176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.624069:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076786176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.624077:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.624080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.624083:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767587.624090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767587.624111:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767587.624123:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767587/real 1775767587] req@ffffa04349640700 x1862023076786304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767587.624132:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076786304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767587.624138:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767587.624140:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767587.624142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076786304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767587.815136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767587.815174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767588.647163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767588.647225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767588.839108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767588.839108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767589.671168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767589.671222:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767589.863111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767589.863112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767590.695162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767590.695229:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767590.887128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767590.887143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767591.719142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767591.719180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767591.911133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767591.911139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767592.743120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767592.743121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767592.743130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767592.743134:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743143:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743151:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775767592.743154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767592.743158:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767592.743194:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076786432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767592.743221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767592.743225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767592.743227:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743232:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743237:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743242:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767592.743251:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767592.743260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076786432:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767592.743266:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076786560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767592.743275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767592.743277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767592.743278:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743281:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743284:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743287:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767592.743307:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767592.743317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767592.743321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076786688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767592.743333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767592.743336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767592.743338:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743343:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743348:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743354:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743354:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7f480 x1862023076786432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767592.743365:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767592.743376:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076786432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767592.743382:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076786816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767592.743396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767592.743400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767592.743404:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767592.743406:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767592.743410:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767592.743413:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076786432:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767592.743417:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743423:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743427:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767592.743431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767592.743440:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076786944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767592.743445:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767592.743453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767592.743455:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1775767592.743471:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767592.743472:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743478:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743482:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767592.743486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7e680 x1862023076786560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767592.743488:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767592.743496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767592.743502:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076786560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767592.743509:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076787072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767592.743511:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743518:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767592.743524:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767592.743524:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767592.743531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767592.743533:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767592.743535:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767592.743539:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743542:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743544:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767592.743548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767592.743556:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767592.743559:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076787200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767592.743566:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767592.743566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7c380 x1862023076786688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767592.743574:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767592.743578:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767592.743587:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00100000:3.0:1775767592.743590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076786688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767592.743591:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767592.743598:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767592.743605:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767592.743606:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767592.743609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767592.743621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767592.743625:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076787328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767592.743635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767592.743639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767592.743643:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767592.743646:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767592.743649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7dc00 x1862023076786816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767592.743654:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4094 00000100:00080000:2.0:1775767592.743664:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767592.743664:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076786816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767592.743673:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767592.743675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743697:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767592.743699:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767592.743701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767592.743720:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076787456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767592.743721:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767592.743751:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7c700 x1862023076786944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767592.743779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076786944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767592.743788:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076786944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767592.743799:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767592.743815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7c000 x1862023076787072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767592.743830:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076787072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767592.743835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767592.743843:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767592.743853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7f100 x1862023076787200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767592.743865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076787200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767592.743870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743874:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767592.743877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767592.743885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743890:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7f480 x1862023076787328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767592.743897:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076787328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767592.743901:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743914:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767592.743924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767592.743936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767592.743942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767592/real 1775767592] req@ffffa0434aa7e300 x1862023076787456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767592.743950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076787456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767592.743955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767592.743957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767592.743959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076787456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767592.935101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767592.935109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767593.767198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767593.767276:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767593.959102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767593.959104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767594.791123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767594.791160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767594.983127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767594.983127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767595.815121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767595.815200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767596.007113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767596.007115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767596.839152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767596.839155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767597.031132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767597.031132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767597.863142:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863156:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767597.863163:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863176:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863204:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775767597.863208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767597.863209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767597.863213:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767597.863286:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076787584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767597.863362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863365:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863369:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863372:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863380:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767597.863390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076787712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863397:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863399:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767597.863400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863403:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863405:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863407:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863410:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767597.863419:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076787840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863426:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863430:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767597.863430:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076787584:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767597.863433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863440:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863448:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863451:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863456:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767597.863467:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076787968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863474:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863476:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767597.863478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863480:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863483:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863485:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863487:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767597.863491:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767597.863497:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076788096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863504:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863507:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767597.863508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863511:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863513:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863516:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863519:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767597.863529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076788224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767597.863529:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7d500 x1862023076787584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.863566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076787584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767597.863577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863584:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767597.863585:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767597.863586:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767597.863590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767597.863596:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00100000:3.0:1775767597.863596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076787584:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767597.863624:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767597.863626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767597.863689:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863695:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767597.863701:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767597.863709:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076788352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767597.863715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7fb80 x1862023076787712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767597.863723:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863730:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767597.863732:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767597.863735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076787712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767597.863738:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00080000:2.0:1775767597.863744:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863747:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767597.863747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767597.863755:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767597.863755:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.863760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767597.863766:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076788480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767597.863773:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767597.863775:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767597.863776:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767597.863778:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4099 00000100:00100000:3.0:1775767597.863782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767597.863790:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767597.863795:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767597.863800:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767597.863804:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767597.863817:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076788608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767597.863817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7f800 x1862023076787840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.863845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076787840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.863855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.863860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.863864:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767597.863871:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767597.863891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.863904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7ed80 x1862023076787968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.863916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076787968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.863925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.863929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.863933:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076787968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767597.863939:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767597.863958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.863969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7ce00 x1862023076788096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.863982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076788096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.863991:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.863995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.863998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767597.864004:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767597.864043:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.864090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7ea00 x1862023076788224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.864110:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076788224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.864123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.864127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.864133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767597.864141:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767597.864169:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.864199:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7df80 x1862023076788352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.864212:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076788352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.864221:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.864224:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.864228:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767597.864234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767597.864272:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.864287:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7d880 x1862023076788480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.864304:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076788480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.864316:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.864321:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.864325:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767597.864368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767597.864389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767597.864400:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767597/real 1775767597] req@ffffa0434aa7fb80 x1862023076788608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767597.864412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076788608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767597.864422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767597.864425:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767597.864428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076788608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767598.055096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767598.055100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767598.887143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767598.887170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767599.079124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767599.079124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767599.911159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767599.911214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767600.103100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767600.103123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767600.935135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767600.935176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767601.127135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767601.127135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767601.959115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767601.959182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767602.151091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767602.151109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767602.983155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767602.983155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767602.983164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767602.983168:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983176:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983185:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767602.983206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767602.983210:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767602.983243:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076788736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767602.983278:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983282:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983288:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983293:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983303:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767602.983312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076788736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767602.983320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076788864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983331:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767602.983333:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983336:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983339:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983341:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767602.983356:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076788992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767602.983372:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983374:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983377:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983380:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983384:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767602.983389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767602.983395:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076789120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983402:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767602.983406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983409:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983412:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983414:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767602.983425:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076789248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983431:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767602.983437:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349641c00 x1862023076788736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767602.983441:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983450:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983457:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983463:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076788736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767602.983470:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767602.983479:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.983485:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.983490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076788736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767602.983493:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076789376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983515:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983517:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767602.983519:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767602.983522:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983526:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767602.983528:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767602.983532:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983541:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767602.983551:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076789504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767602.983559:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983562:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767602.983565:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767602.983566:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767602.983570:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983575:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767602.983578:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767602.983582:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767602.983608:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349641180 x1862023076788864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767602.983621:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076789632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767602.983628:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076788864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767602.983629:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767602.983636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767602.983638:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767602.983646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767602.983648:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4104 00000100:00080000:0.0:1775767602.983654:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767602.983655:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767602.983660:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983664:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788864:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767602.983667:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767602.983677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767602.983680:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076789760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767602.983711:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349643b80 x1862023076788992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.983737:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076788992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.983746:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.983749:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.983753:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076788992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767602.983759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076789120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767602.983780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983792:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349643800 x1862023076789120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.983804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076789120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.983813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.983817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.983820:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076789120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767602.983827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076789248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767602.983846:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349641880 x1862023076789248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.983868:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076789248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.983877:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.983880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.983884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076789248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767602.983890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767602.983906:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349642680 x1862023076789376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.983927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076789376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.983936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.983940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.983943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767602.983965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767602.983982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.983992:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349641c00 x1862023076789504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.984004:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076789504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.984012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.984016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.984019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767602.984024:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767602.984065:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.984076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349640700 x1862023076789632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.984087:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076789632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.984094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.984097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.984100:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767602.984107:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767602.984125:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767602.984135:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767602/real 1775767602] req@ffffa04349640e00 x1862023076789760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767602.984145:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076789760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767602.984153:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767602.984157:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767602.984159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076789760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767603.175119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767603.175119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767604.007128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767604.007150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767604.199116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767604.199116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767605.031120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767605.031168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767605.223121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767605.223217:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767606.055126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767606.055339:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767606.247123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767606.247123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775767607.079138:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767607.079146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767607.271120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767607.271120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767608.103112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767608.103128:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103136:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767608.103140:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103151:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4109 00010000:00010000:3.0:1775767608.103162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767608.103163:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103172:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767608.103208:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076789888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103229:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103232:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767608.103233:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103236:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103240:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103245:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103254:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767608.103259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076789888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767608.103273:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076790016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103306:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767608.103309:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767608.103314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767608.103315:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103321:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103324:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767608.103341:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076790144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103348:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103350:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767608.103352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103355:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103357:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767608.103361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349641500 x1862023076789888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767608.103362:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103369:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767608.103379:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076790272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103386:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767608.103391:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767608.103392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076789888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767608.103397:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:3.0:1775767608.103404:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767608.103406:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103411:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767608.103414:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775767608.103416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767608.103421:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076789888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767608.103427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076790400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103435:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103437:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767608.103438:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103441:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103444:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103446:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767608.103458:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076790528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767608.103460:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790016:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767608.103477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767608.103480:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767608.103481:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103485:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103499:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767608.103506:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767608.103507:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103515:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767608.103523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349640a80 x1862023076790016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767608.103530:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076790656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767608.103563:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076790016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103596:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767608.103597:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767608.103602:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767608.103605:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767608.103607:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790016:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767608.103608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103617:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00100000:3.0:1775767608.103619:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767608.103623:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103627:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767608.103632:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767608.103643:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767608.103646:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076790784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767608.103654:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767608.103656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349642300 x1862023076790144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767608.103657:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767608.103660:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767608.103667:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4109 00000100:00080000:0.0:1775767608.103673:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767608.103676:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076790144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767608.103681:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767608.103691:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076790912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767608.103698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103711:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767608.103718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767608.103741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349643100 x1862023076790272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076790272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767608.103777:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767608.103788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349640380 x1862023076790400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076790400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076790400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767608.103822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767608.103833:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349642a00 x1862023076790528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076790528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103865:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767608.103873:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767608.103884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349641500 x1862023076790656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076790656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103901:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103903:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767608.103908:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767608.103917:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349640e00 x1862023076790784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076790784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103935:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767608.103941:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767608.103950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767608.103954:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767608/real 1775767608] req@ffffa04349640700 x1862023076790912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767608.103961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076790912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767608.103966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767608.103967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767608.103969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076790912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767608.295106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767608.295107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767609.127137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767609.127214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767609.319137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767609.319137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767610.151122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767610.151146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767610.343104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767610.343104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767611.175133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767611.175219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767611.367143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767611.367143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767612.199131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767612.199164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767612.391124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767612.391131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767613.223125:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223133:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775767613.223140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767613.223151:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223158:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223168:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223175:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767613.223205:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076791040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223223:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223226:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767613.223228:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223232:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223235:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223238:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223246:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767613.223248:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076791040:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767613.223259:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076791168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223269:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223272:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767613.223273:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223276:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223279:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223282:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767613.223298:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076791296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767613.223306:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767613.223318:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223321:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767613.223323:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223327:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223331:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223343:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767613.223353:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7d500 x1862023076791040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767613.223364:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076791424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223372:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223374:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767613.223375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223377:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223379:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223381:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767613.223389:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076791552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223395:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767613.223398:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223400:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223406:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223409:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767613.223418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076791680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223424:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223426:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767613.223427:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223430:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223431:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223433:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767613.223441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076791808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767613.223446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223448:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767613.223450:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223452:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223454:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223456:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223460:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767613.223470:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076791936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767613.223470:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076791040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767613.223502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767613.223504:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767613.223505:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767613.223508:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4114 00000100:00080000:2.0:1775767613.223511:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767613.223514:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767613.223518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767613.223526:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076792064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767613.223586:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076791040:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767613.223651:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767613.223676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223687:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7e300 x1862023076791168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076791168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223704:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223706:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767613.223710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767613.223721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223727:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7f480 x1862023076791296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076791296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223739:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223741:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767613.223746:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767613.223757:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7f100 x1862023076791424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223769:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076791424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223774:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767613.223780:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767613.223789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7c000 x1862023076791552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076791552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076791552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767613.223813:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767613.223822:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7c700 x1862023076791680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076791680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223846:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767613.223849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767613.223858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7dc00 x1862023076791808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223869:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076791808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767613.223881:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767613.223890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223895:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7c380 x1862023076791936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223901:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076791936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223920:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076791936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767613.223926:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767613.223937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767613.223942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767613/real 1775767613] req@ffffa0434aa7e680 x1862023076792064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767613.223949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076792064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767613.223954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767613.223956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767613.223957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792064:192.168.202.158@tcp:8:kworker.0 00010000:00010000:0.0:1775767613.224049:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767613.415136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767613.415136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767614.247127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767614.247152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767614.439105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767614.439109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767615.271143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767615.271172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767615.463120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767615.463120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767616.295145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767616.295189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767616.487173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767616.487175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775767617.319199:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767617.319217:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767617.511148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767617.511151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767618.343116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767618.343117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767618.343128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767618.343135:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343147:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343158:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343165:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767618.343209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767618.343211:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076792192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767618.343261:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343265:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343275:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767618.343277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076792192:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767618.343286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767618.343298:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076792320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767618.343310:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343313:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343316:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343318:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343322:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767618.343333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076792448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767618.343349:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343353:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343358:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767618.343363:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767618.343368:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767618.343392:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076792576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767618.343404:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343407:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343410:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343413:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343418:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767618.343427:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7e680 x1862023076792192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767618.343431:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076792704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343473:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767618.343481:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343487:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767618.343492:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076792192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767618.343497:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343505:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767618.343510:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767618.343517:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076792832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767618.343519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343530:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076792192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767618.343531:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767618.343542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343553:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343557:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343560:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343565:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767618.343568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767618.343575:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076792960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767618.343583:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343586:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767618.343588:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343591:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343594:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343598:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767618.343600:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343602:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767618.343612:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076793088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767618.343619:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7c380 x1862023076792320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767618.343620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767618.343628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767618.343630:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767618.343638:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4120 00000100:00080000:2.0:1775767618.343642:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767618.343646:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767618.343650:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767618.343654:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076792320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767618.343659:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076793216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767618.343670:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.343677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343682:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767618.343691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767618.343715:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.343727:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7dc00 x1862023076792448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.343741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076792448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.343757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.343760:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767618.343770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767618.343790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.343801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7c700 x1862023076792576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.343813:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076792576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.343822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.343826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767618.343835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767618.343853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.343863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7c000 x1862023076792704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.343875:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076792704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.343894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.343897:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076792704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767618.343907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767618.343928:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.343939:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7f100 x1862023076792832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.343952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076792832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.343961:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.343964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.343968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767618.343985:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767618.344003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.344013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7f480 x1862023076792960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.344044:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076792960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.344054:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.344057:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.344061:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076792960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767618.344067:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767618.344085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.344097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7e300 x1862023076793088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.344109:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076793088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.344118:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.344121:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.344125:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767618.344130:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767618.344147:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767618.344156:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767618/real 1775767618] req@ffffa0434aa7d500 x1862023076793216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767618.344168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076793216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767618.344177:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767618.344181:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767618.344184:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767618.535176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767618.535176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767619.367182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767619.367249:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767619.559114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767619.559122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767620.391096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767620.391134:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767620.583142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767620.583143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767621.415165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767621.415256:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767621.607147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767621.607147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767622.439140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767622.439207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767622.631150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767622.631150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767623.463120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767623.463129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767623.463132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767623.463139:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463157:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463174:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463185:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767623.463187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767623.463240:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076793344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463268:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767623.463271:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463275:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463280:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463284:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463287:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076793344:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767623.463296:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767623.463308:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076793472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463337:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767623.463339:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463343:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463347:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463351:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767623.463357:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767623.463364:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767623.463373:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076793600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463382:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767623.463402:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463405:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463409:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463412:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767623.463418:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767623.463427:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7fb80 x1862023076793344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767623.463428:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076793728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767623.463444:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463447:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00100000:3.0:1775767623.463448:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076793344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767623.463451:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463455:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767623.463459:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463464:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767623.463467:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767623.463468:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076793344:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767623.463476:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076793856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463488:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767623.463490:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767623.463493:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767623.463495:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463499:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463503:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767623.463508:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767623.463515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767623.463517:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076793984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767623.463527:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767623.463527:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7d880 x1862023076793472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767623.463530:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767623.463532:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463537:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463540:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767623.463540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076793472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767623.463544:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767623.463548:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767623.463549:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463553:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463556:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767623.463557:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076794112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767623.463563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793600:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767623.463567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767623.463581:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463590:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7df80 x1862023076793600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767623.463597:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463601:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4125 00000100:00080000:2.0:1775767623.463605:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463609:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076793600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767623.463615:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767623.463617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767623.463624:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076794240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767623.463624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767623.463630:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793728:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767623.463635:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767623.463638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767623.463640:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767623.463644:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4125 00000800:00000100:3.0:1775767623.463647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767623.463648:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767623.463652:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767623.463657:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767623.463657:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7ea00 x1862023076793728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767623.463666:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076794368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767623.463668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076793728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767623.463689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767623.463707:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463718:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7ce00 x1862023076793856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767623.463725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076793856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463736:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076793856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767623.463754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767623.463766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463773:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7ed80 x1862023076793984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767623.463781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076793984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463786:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463788:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076793984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767623.463793:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767623.463803:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7f800 x1862023076794112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767623.463816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076794112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463825:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767623.463828:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767623.463838:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7fb80 x1862023076794240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767623.463850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076794240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767623.463861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767623.463869:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767623.463874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767623/real 1775767623] req@ffffa0434aa7d500 x1862023076794368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767623.463881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076794368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767623.463886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767623.463888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767623.463890:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076794368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767623.655123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767623.655136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767624.487143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767624.487245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767624.679123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767624.679124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767625.511108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767625.511134:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767625.703119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767625.703119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767626.535141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767626.535211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767626.727137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767626.727137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767627.559165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767627.559166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767627.751137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767627.751137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767628.583142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767628.583215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583221:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767628.583225:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583234:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583243:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583249:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767628.583275:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767628.583284:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076794496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767628.583306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583308:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767628.583309:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583312:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583315:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583348:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076794496:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775767628.583348:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767628.583360:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076794624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767628.583375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767628.583379:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583382:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583385:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583388:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767628.583391:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767628.583399:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076794752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767628.583406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583408:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767628.583410:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583412:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583414:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583417:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767628.583420:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767628.583426:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076794880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767628.583433:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767628.583438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583441:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767628.583442:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583445:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583448:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583451:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767628.583454:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767628.583460:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076795008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767628.583466:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583468:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767628.583469:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583471:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583473:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583476:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767628.583479:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767628.583487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825ce00 x1862023076794496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767628.583488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076795136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767628.583521:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583525:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767628.583527:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767628.583529:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076794496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767628.583535:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:2.0:1775767628.583543:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767628.583543:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583550:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767628.583551:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583558:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076794496:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775767628.583559:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767628.583574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076795264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767628.583580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767628.583592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767628.583596:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767628.583599:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767628.583604:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4130 00000800:00000100:2.0:1775767628.583608:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767628.583611:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583617:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583622:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825d880 x1862023076794624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767628.583624:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767628.583638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076794624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767628.583642:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076795392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767628.583653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583661:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767628.583663:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767628.583666:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767628.583671:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767628.583674:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767628.583676:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767628.583682:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4130 00000100:00080000:3.0:1775767628.583686:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767628.583690:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767628.583695:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:3.0:1775767628.583696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767628.583704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825dc00 x1862023076794752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767628.583711:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076795520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767628.583738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076794752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583746:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583749:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767628.583766:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767628.583782:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825d500 x1862023076794880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.583801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076794880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076794880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767628.583813:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767628.583823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825f100 x1862023076795008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.583835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076795008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767628.583856:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767628.583869:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583875:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825e300 x1862023076795136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.583882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076795136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767628.583903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767628.583913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825ed80 x1862023076795264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.583925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076795264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767628.583943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767628.583954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825c000 x1862023076795392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.583968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076795392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.583973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.583974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.583976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767628.583979:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767628.583989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767628.583993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767628/real 1775767628] req@ffffa0435825e680 x1862023076795520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767628.584000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076795520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767628.584005:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767628.584007:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767628.584009:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076795520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767628.775143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767628.775143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767629.607156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767629.607161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767629.799111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767629.799111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767630.631087:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767630.631204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767630.823105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767630.823113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767631.655138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767631.655219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767631.847093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767631.847100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767632.679135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767632.679180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767632.871141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767632.871141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775767633.703153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767633.703156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767633.703170:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703180:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703192:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703199:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767633.703237:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076795648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767633.703263:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703266:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703270:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703282:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767633.703308:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076795776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767633.703320:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076795648:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767633.703321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767633.703332:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703337:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703341:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703345:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703350:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767633.703361:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076795904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703369:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703371:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767633.703373:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703376:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703379:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703382:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703385:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767633.703393:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076796032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767633.703403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703407:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703408:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703411:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703415:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767633.703417:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767633.703426:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076796160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767633.703439:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703441:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703445:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767633.703448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795904:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767633.703452:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703457:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767633.703469:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076796288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767633.703472:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767633.703475:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796032:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767633.703488:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767633.703500:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767633.703501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796160:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767633.703507:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703512:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703514:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767633.703525:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767633.703530:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076796416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703538:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703541:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767633.703542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767633.703545:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00100000:2.0:1775767633.703548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767633.703548:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703554:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703559:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767633.703570:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076796544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767633.703585:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767633.703588:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767633.703590:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767633.703595:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796544:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767633.703595:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4135 00000100:00080000:3.0:1775767633.703603:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767633.703606:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767633.703610:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767633.703621:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076796672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767633.703643:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767633.895099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767633.895099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767634.727130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767634.727176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767634.919101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767634.919107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767635.751101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767635.751121:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767635.943129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767635.943129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767636.775151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767636.775155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775767636.775401:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775767636.775442:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767636.775475:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775524:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775537:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775545:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775552:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775560:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775567:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775573:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767636.775580:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767636.775763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825d880 x1862023076796672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076796672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775803:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.775815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767636.775825:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825dc00 x1862023076796544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076796544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775852:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.775855:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767636.775862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825d500 x1862023076796416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076796416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.775897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767636.775904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825e300 x1862023076796160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076796160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.775934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767636.775941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825f100 x1862023076796288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775952:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076796288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775964:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.775967:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076796288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767636.775974:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825e680 x1862023076795776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.775986:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076795776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.775994:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.775997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.776000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767636.776006:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825c000 x1862023076795904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.776016:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076795904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.776023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.776142:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.776146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076795904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767636.776155:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825ed80 x1862023076796032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.776175:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076796032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.776183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.776187:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.776190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767636.776211:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767633/real 1775767636] req@ffffa0435825ea00 x1862023076795648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767636.776227:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076795648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767636.776234:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767636.776238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767636.776241:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076795648:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767636.967111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767636.967111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767637.799120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767637.799179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767637.991130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767637.991130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767638.823146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767638.823234:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1775767638.823235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767638.823257:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767638.823263:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823277:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823289:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823297:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767638.823333:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076796800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823368:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767638.823370:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823375:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823379:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823385:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823396:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767638.823420:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076796928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767638.823420:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076796800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767638.823448:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823452:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767638.823472:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823479:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823498:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767638.823513:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076797056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823527:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823531:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767638.823538:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767638.823538:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767638.823553:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823560:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823566:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823573:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767638.823589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076797184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823601:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823604:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767638.823607:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823613:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823618:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767638.823620:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349641c00 x1862023076796800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767638.823627:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823641:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767638.823659:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076796800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767638.823672:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076797312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767638.823679:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.823695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767638.823697:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767638.823704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076796800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767638.823706:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767638.823711:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823719:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823725:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823732:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823739:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767638.823748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767638.823758:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076797440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823776:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823783:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775767638.823784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767638.823796:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767638.823806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349642680 x1862023076796928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767638.823807:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823820:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823828:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767638.823833:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076796928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767638.823838:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767638.823859:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767638.823872:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076797568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823885:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767638.823888:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767638.823890:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767638.823904:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823917:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767638.823920:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.823927:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767638.823941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076797696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767638.823950:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767638.823956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767638.823957:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767638.823967:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767638.823972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076796928:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767638.823981:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4140 00000100:00080000:0.0:1775767638.823987:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767638.823990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797056:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767638.823999:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767638.824005:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767638.824023:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076797824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767638.824057:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824073:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349641880 x1862023076797056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076797056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824103:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767638.824120:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767638.824145:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824163:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349643800 x1862023076797184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076797184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824188:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824193:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767638.824208:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767638.824226:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824253:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349643b80 x1862023076797312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076797312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824276:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824284:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076797312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767638.824308:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767638.824329:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824339:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349641180 x1862023076797440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824351:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076797440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824368:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767638.824374:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767638.824390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824414:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349641c00 x1862023076797568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824426:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076797568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824440:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824445:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767638.824453:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767638.824479:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349640700 x1862023076797696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076797696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824515:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824518:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824522:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767638.824529:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767638.824546:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767638.824555:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767638/real 1775767638] req@ffffa04349640e00 x1862023076797824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767638.824566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076797824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767638.824576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767638.824580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767638.824583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076797824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767639.015100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767639.015105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767639.847118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767639.847167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767640.039150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767640.039150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767640.871158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767640.871174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767641.063104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767641.063162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767641.895198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767641.895221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767642.087117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767642.087144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767642.919162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767642.919164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767643.111121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767643.111152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767643.943157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767643.943157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767643.943170:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943181:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943206:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767643.943212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767643.943214:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767643.943256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076797952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767643.943284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767643.943305:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943311:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943316:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767643.943321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076797952:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767643.943325:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767643.943365:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076798080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767643.943374:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943377:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767643.943379:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943382:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943385:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767643.943389:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943395:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767643.943395:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767643.943409:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076798208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767643.943420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943424:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767643.943426:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943431:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943436:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767643.943439:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350930700 x1862023076797952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767643.943445:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943456:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767643.943462:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076797952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.943475:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.943484:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076797952:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767643.943485:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076798336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767643.943504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767643.943511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943516:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767643.943519:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943525:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4145 00000800:00000100:3.0:1775767643.943528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767643.943534:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767643.943539:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943544:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350932a00 x1862023076798080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767643.943546:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767643.943558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076798080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767643.943563:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076798464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767643.943572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767643.943578:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767643.943580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767643.943585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767643.943588:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767643.943589:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943597:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00100000:3.0:1775767643.943600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798208:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767643.943602:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767643.943608:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943612:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767643.943620:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767643.943623:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076798592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767643.943629:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350932300 x1862023076798208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767643.943636:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943644:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767643.943646:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767643.943649:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076798208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.943660:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767643.943661:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:3.0:1775767643.943666:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767643.943667:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767643.943670:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798208:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767643.943674:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943680:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767643.943686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767643.943691:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076798720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767643.943703:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767643.943707:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775767643.943713:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350930e00 x1862023076798336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767643.943714:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943723:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943729:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767643.943732:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076798336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767643.943737:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767643.943747:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767643.943753:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076798848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767643.943757:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767643.943769:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798464:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767643.943770:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767643.943776:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767643.943777:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767643.943781:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4145 00000100:00080000:1.0:1775767643.943785:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767643.943788:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767643.943791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767643.943798:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767643.943802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350931880 x1862023076798464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767643.943813:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076798464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767643.943817:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076798976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767643.943821:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.943833:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076798464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767643.943844:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767643.943861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350930000 x1862023076798592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767643.943876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076798592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.943881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943883:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.943885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767643.943894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767643.943915:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350930a80 x1862023076798720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767643.943930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076798720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.943935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.943939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767643.943942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767643.943952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350931c00 x1862023076798848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767643.943965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076798848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.943969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.943971:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.943973:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767643.943976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767643.943985:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767643.943989:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767643/real 1775767643] req@ffffa04350932680 x1862023076798976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767643.943996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076798976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767643.944001:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767643.944003:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767643.944005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076798976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767644.135174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767644.135174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767644.967138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767644.967173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767645.159109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767645.159161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767645.991134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767645.991251:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767646.183109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767646.183154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767647.015177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767647.015245:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767647.207113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767647.207113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767648.039146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767648.039264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767648.231123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767648.231123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767649.063126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767649.063286:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1775767649.063291:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767649.063292:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767649.063308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063316:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063325:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063331:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767649.063360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076799104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767649.063384:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063387:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063390:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063393:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063401:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767649.063413:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076799232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767649.063415:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076799104:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767649.063429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063432:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767649.063434:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063439:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063443:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063447:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063453:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767649.063465:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076799360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063488:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767649.063494:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063498:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063503:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063507:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063513:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767649.063523:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076799488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767649.063534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063540:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063542:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063546:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767649.063547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767649.063558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076799616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063565:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063567:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767649.063569:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063571:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063579:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063582:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063585:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767649.063592:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076799744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063599:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063601:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767649.063602:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063604:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063605:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063607:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063609:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767649.063615:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076799872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063621:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063623:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767649.063624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063629:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063631:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063633:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767649.063640:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076800000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767649.063654:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767649.063656:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767649.063658:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767649.063660:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4150 00000100:00080000:2.0:1775767649.063663:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767649.063665:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767649.063668:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767649.063677:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076800128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767649.063742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7e300 x1862023076799104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076799104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.063767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.063770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.063773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076799104:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767649.063796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767649.063815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.063832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7f480 x1862023076799232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076799232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.063847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.063850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.063852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767649.063856:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767649.063868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.063874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7f100 x1862023076799360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076799360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.063887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.063889:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.063891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767649.063895:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767649.063905:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.063910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7c000 x1862023076799488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076799488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.063922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.063923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.063932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767649.063936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767649.063947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.063953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7c700 x1862023076799616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076799616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.063966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.063968:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.063970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076799616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767649.063973:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076799744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767649.063983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.063988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7dc00 x1862023076799744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.063995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076799744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.064000:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.064002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.064004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076799744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767649.064007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076799872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767649.064016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.064021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7c380 x1862023076799872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.064051:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076799872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.064057:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.064060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.064062:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076799872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767649.064068:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767649.064081:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.064093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7e680 x1862023076800000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.064104:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076800000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.064112:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.064115:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.064118:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767649.064124:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767649.064157:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767649.064169:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767649/real 1775767649] req@ffffa0434aa7d500 x1862023076800128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767649.064181:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076800128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767649.064191:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767649.064194:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767649.064197:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767649.255106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767649.255134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767650.087112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767650.087201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767650.279141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767650.279141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767651.111165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767651.111255:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767651.303089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767651.303099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767652.135184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767652.135213:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767652.327150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767652.327150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767653.159118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767653.159170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767653.351088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767653.351125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767654.183153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767654.183226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767654.183235:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183242:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183250:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183256:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767654.183288:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076800256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183314:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767654.183318:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183321:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183323:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183326:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183334:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767654.183342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076800384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183353:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767654.183354:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183357:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183359:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183362:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183365:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767654.183370:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076800256:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767654.183375:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076800512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767654.183388:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183391:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183392:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183395:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183398:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767654.183405:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076800640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183413:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767654.183415:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183417:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183420:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183422:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183426:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767654.183432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076800768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183438:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1775767654.183445:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767654.183445:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183452:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183454:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183456:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183466:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767654.183473:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076800896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183479:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767654.183483:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183485:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183488:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183492:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825ea00 x1862023076800256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767654.183496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767654.183505:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076801024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767654.183523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767654.183526:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767654.183527:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076800256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767654.183528:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183534:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183538:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767654.183544:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183549:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767654.183553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767654.183557:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076801152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767654.183562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767654.183568:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767654.183572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076800256:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767654.183573:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767654.183575:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767654.183581:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4155 00000100:00080000:3.0:1775767654.183584:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767654.183587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767654.183588:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767654.183594:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767654.183602:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076801280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767654.183623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183640:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825ed80 x1862023076800384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183657:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076800384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183667:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183685:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00010000:00010000:1.0:1775767654.183689:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767654.183691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767654.183720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767654.183742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183751:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825c000 x1862023076800512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076800512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183765:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.183772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767654.183778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767654.183797:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825e680 x1862023076800640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076800640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183829:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183832:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.183836:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767654.183844:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767654.183875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825f100 x1862023076800768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183898:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076800768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183908:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.183910:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076800768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767654.183914:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767654.183928:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825e300 x1862023076800896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076800896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.183950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076800896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767654.183954:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767654.183964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.183969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825d500 x1862023076801024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.183986:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076801024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.183991:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.183993:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.183995:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767654.183999:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767654.184010:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.184016:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825dc00 x1862023076801152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.184023:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076801152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.184046:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.184048:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.184051:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767654.184056:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767654.184076:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767654.184087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767654/real 1775767654] req@ffffa0435825d880 x1862023076801280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767654.184112:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076801280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767654.184127:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767654.184130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767654.184132:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076801280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767654.375129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767654.375139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767655.207176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767655.207198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767655.399159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767655.399172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767656.231134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767656.231155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767656.423140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767656.423154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767657.255119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767657.255140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767657.447137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767657.447137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767658.279110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767658.279173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767658.471121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767658.471121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767659.303132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767659.303146:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767659.303152:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767659.303156:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303164:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303173:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775767659.303179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767659.303179:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767659.303223:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076801408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303244:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767659.303247:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767659.303261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303268:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303272:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076801408:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767659.303282:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767659.303296:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076801536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303308:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767659.303312:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767659.303314:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303319:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303323:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303328:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767659.303331:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767659.303336:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767659.303352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076801664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767659.303368:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7fb80 x1862023076801408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767659.303369:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767659.303372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303380:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303385:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303389:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076801408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767659.303395:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767659.303402:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303409:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303412:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076801408:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767659.303418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076801792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303426:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767659.303428:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767659.303430:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767659.303434:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767659.303435:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303438:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303441:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767659.303447:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767659.303457:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076801920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303465:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767659.303470:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767659.303470:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767659.303473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303479:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303482:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303486:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7f800 x1862023076801536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767659.303490:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767659.303504:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076802048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767659.303505:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076801536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767659.303529:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767659.303531:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767659.303532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767659.303536:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767659.303538:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303540:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767659.303549:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303553:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767659.303556:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303559:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801664:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767659.303560:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767659.303575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076802176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767659.303581:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767659.303583:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767659.303587:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767659.303589:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767659.303593:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303596:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767659.303598:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7ed80 x1862023076801664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767659.303602:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767659.303609:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767659.303616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076801664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767659.303620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076802304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767659.303636:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767659.303638:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767659.303642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767659.303642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767659.303645:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767659.303649:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767659.303653:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4160 00000100:00080000:2.0:1775767659.303657:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767659.303660:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801792:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767659.303661:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767659.303668:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767659.303679:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076802432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767659.303685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7ce00 x1862023076801792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303740:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076801792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767659.303775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767659.303793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7ea00 x1862023076801920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076801920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076801920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767659.303822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767659.303832:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7df80 x1862023076802048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076802048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303859:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303861:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767659.303867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767659.303879:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7d880 x1862023076802176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076802176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303899:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767659.303904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767659.303913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7fb80 x1862023076802304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076802304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303932:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767659.303942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767659.303955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767659.303961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767659/real 1775767659] req@ffffa0434aa7d500 x1862023076802432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767659.303968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076802432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767659.303973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767659.303975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767659.303977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076802432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767659.495175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767659.495175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767660.327153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767660.327212:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767660.519116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767660.519120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767661.351123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767661.351172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767661.543132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767661.543132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767662.375115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767662.375169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767662.567125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767662.567128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767663.399144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767663.399179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767663.591125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767663.591125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767664.423158:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767664.423170:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775767664.423174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767664.423176:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775767664.423184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767664.423196:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423207:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423216:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767664.423278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076802560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767664.423315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767664.423317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767664.423323:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423331:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423348:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.423360:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767664.423376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076802560:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767664.423382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076802688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423404:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767664.423409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767664.423411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767664.423417:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423422:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423427:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.423433:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767664.423449:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076802816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767664.423467:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767664.423469:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767664.423481:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767664.423506:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423521:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.423529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767664.423545:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076802944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423557:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767664.423560:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349641500 x1862023076802560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767664.423567:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767664.423574:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767664.423595:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423602:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423608:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.423615:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767664.423621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076802560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767664.423635:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076803072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767664.423647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767664.423655:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767664.423661:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767664.423664:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767664.423671:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767664.423675:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076802560:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767664.423685:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423693:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423698:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.423705:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767664.423716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767664.423723:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076803200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423749:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767664.423750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767664.423761:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767664.423764:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767664.423772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349642a00 x1862023076802688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767664.423773:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423788:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767664.423794:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076802688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767664.423796:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767664.423814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767664.423815:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767664.423822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.423826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767664.423833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767664.423839:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076803328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767664.423852:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767664.423862:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767664.423864:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767664.423872:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767664.423880:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4166 00000100:00080000:0.0:1775767664.423886:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767664.423905:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767664.423907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349640380 x1862023076802816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767664.423913:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767664.423937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076802816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767664.423941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076803456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767664.424012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767664.424020:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767664.424022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767664.424061:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802944:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767664.424070:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767664.424073:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767664.424079:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4166 00000800:00000100:3.0:1775767664.424086:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767664.424097:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767664.424103:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349643100 x1862023076802944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767664.424108:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767664.424126:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767664.424126:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076802944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424144:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424151:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076802944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767664.424156:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076803584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767664.424158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767664.424180:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767664.424195:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349642300 x1862023076803072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767664.424207:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076803072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424213:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424216:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767664.424250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767664.424269:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767664.424295:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349640a80 x1862023076803200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767664.424308:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076803200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424320:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424324:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424328:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767664.424336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767664.424362:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767664.424376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349641500 x1862023076803328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767664.424411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076803328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424425:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424429:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767664.424436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767664.424457:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767664.424469:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349640e00 x1862023076803456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767664.424481:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076803456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424492:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424495:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767664.424500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767664.424532:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767664.424540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767664/real 1775767664] req@ffffa04349640700 x1862023076803584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767664.424555:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076803584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767664.424565:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767664.424570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767664.424574:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076803584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767664.615143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767664.615149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767665.447174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767665.447207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767665.639166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767665.639175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767666.471137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767666.471220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767666.663157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767666.663201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767667.495142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767667.495179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767667.687126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767667.687163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767668.519124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767668.519225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767668.711128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767668.711129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767669.543140:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543149:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767669.543153:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775767669.543153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767669.543165:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4171 00010000:00010000:3.0:1775767669.543165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767669.543176:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543184:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767669.543217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076803712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767669.543239:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543241:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767669.543243:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767669.543246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:0.0:1775767669.543248:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543252:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543259:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767669.543279:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076803712:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767669.543294:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076803840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767669.543303:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543305:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767669.543306:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767669.543310:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:0.0:1775767669.543315:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543320:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767669.543338:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076803968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767669.543348:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767669.543351:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543355:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767669.543358:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767669.543371:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:0.0:1775767669.543375:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543378:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543385:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767669.543391:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349641c00 x1862023076803712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767669.543400:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076804096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767669.543415:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767669.543419:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076803712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767669.543421:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767669.543425:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767669.543435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767669.543437:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:3.0:1775767669.543441:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767669.543442:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767669.543447:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076803712:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767669.543448:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543457:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767669.543468:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076804224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767669.543475:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767669.543477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543483:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767669.543484:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767669.543488:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:0.0:1775767669.543490:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767669.543502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349641180 x1862023076803840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767669.543519:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076804352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767669.543529:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076803840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767669.543538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767669.543540:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767669.543543:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767669.543546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767669.543547:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767669.543550:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767669.543558:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00100000:3.0:1775767669.543559:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767669.543564:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543569:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767669.543578:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767669.543588:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076804480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767669.543588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349643b80 x1862023076803968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076803968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767669.543609:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543617:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767669.543622:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543624:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767669.543627:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767669.543627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076803968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767669.543635:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00100000:3.0:1775767669.543638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804096:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767669.543641:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543650:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767669.543660:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076804608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767669.543666:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767669.543672:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767669.543674:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767669.543676:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767669.543679:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349643800 x1862023076804096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767669.543680:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4171 00000100:00080000:0.0:1775767669.543685:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767669.543691:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767669.543697:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767669.543700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076804096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767669.543706:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076804736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767669.543708:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.543723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767669.543739:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767669.543752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543758:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349641880 x1862023076804224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076804224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767669.543771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.543774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767669.543778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767669.543787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543792:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349642680 x1862023076804352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076804352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767669.543805:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.543809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767669.543815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767669.543831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349641c00 x1862023076804480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076804480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767669.543872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.543878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767669.543882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767669.543897:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349640700 x1862023076804608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076804608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767669.543922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.543924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.543926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767669.543931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767669.543970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767669.543982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767669/real 1775767669] req@ffffa04349640e00 x1862023076804736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767669.543995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076804736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767669.544003:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767669.544005:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767669.544007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076804736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767669.735098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767669.735098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767670.567125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767670.567197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767670.759102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767670.759102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767671.591127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767671.591173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767671.783121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767671.783122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767672.615145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767672.615271:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767672.807128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767672.807138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767673.639150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767673.639235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767673.831151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767673.831151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767674.663118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767674.663118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767674.663129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767674.663134:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663144:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663155:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775767674.663158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767674.663163:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767674.663201:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076804864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767674.663229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767674.663232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767674.663234:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663240:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663245:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767674.663250:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076804864:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767674.663260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767674.663277:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076804992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767674.663288:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767674.663292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767674.663294:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663299:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663304:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767674.663308:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767674.663327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767674.663331:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767674.663347:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076805120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767674.663359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767674.663364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349641500 x1862023076804864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663365:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767674.663369:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663378:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663384:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767674.663387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076804864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663392:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767674.663400:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767674.663402:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767674.663407:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663410:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076804864:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767674.663420:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076805248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767674.663433:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767674.663433:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767674.663442:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767674.663444:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663449:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663454:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767674.663460:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767674.663460:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767674.663468:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767674.663473:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349640a80 x1862023076804992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767674.663484:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076805376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767674.663485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076804992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767674.663515:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767674.663520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767674.663520:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076804992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767674.663527:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767674.663531:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767674.663539:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663546:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4176 00000800:00000100:3.0:1775767674.663547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767674.663556:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767674.663557:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349642300 x1862023076805120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663565:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767674.663575:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767674.663580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076805120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767674.663591:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767674.663593:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076805504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767674.663599:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767674.663614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767674.663617:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767674.663619:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767674.663625:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:0.0:1775767674.663630:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767674.663632:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805248:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767674.663637:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767674.663643:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767674.663657:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076805632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767674.663658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349643100 x1862023076805248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663672:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767674.663682:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767674.663685:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767674.663688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076805248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663693:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00080000:3.0:1775767674.663700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767674.663701:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767674.663706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767674.663710:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805248:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767674.663719:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767674.663720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767674.663734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663744:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349640380 x1862023076805376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767674.663746:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076805760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767674.663766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076805376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767674.663773:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767674.663775:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767674.663781:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767674.663781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767674.663786:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767674.663787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076805376:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767674.663796:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4176 00000100:00100000:3.0:1775767674.663797:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805504:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767674.663805:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767674.663812:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767674.663816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767674.663821:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767674.663825:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349642a00 x1862023076805504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767674.663834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076805504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767674.663838:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076805888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767674.663868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767674.663872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767674.663889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767674.663919:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349642300 x1862023076805632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767674.663934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076805632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767674.663940:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767674.663942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767674.663947:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767674.663957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663962:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349640a80 x1862023076805760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767674.663969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076805760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767674.663974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767674.663975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.663978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767674.663982:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767674.663991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767674.663996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767674/real 1775767674] req@ffffa04349641500 x1862023076805888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767674.664003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076805888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767674.664008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767674.664009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767674.664011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076805888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767674.855133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767674.855133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767675.687119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767675.687169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767675.879090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767675.879100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767676.711117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767676.711147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767676.903169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767676.903174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767677.735152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767677.735194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767677.927119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767677.927147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767678.759099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767678.759206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767678.951145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767678.951154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767679.783109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767679.783125:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1775767679.783127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767679.783133:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767679.783137:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783145:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783153:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783158:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767679.783195:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076806016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783216:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783219:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767679.783221:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783223:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783226:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783243:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783248:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076806016:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767679.783253:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767679.783277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076806144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783285:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783288:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767679.783289:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783292:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783296:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783298:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783301:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767679.783306:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767679.783310:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076806272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783319:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783323:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767679.783325:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783328:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783331:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783334:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783338:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767679.783345:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076806400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783352:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767679.783361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767679.783362:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350932680 x1862023076806016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767679.783368:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783373:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783379:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767679.783388:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076806016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767679.783393:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076806528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783407:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767679.783409:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767679.783409:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767679.783412:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783415:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767679.783416:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767679.783420:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076806016:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767679.783426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767679.783436:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076806656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783442:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783444:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767679.783446:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767679.783449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767679.783452:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783458:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783460:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783468:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767679.783477:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767679.783481:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076806784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767679.783490:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767679.783490:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350931c00 x1862023076806144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767679.783497:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767679.783501:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783508:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783512:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767679.783513:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076806144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767679.783516:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767679.783526:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767679.783532:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076806912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767679.783532:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783540:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767679.783547:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767679.783551:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767679.783551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806272:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767679.783554:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767679.783560:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4181 00000100:00080000:1.0:1775767679.783563:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767679.783565:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767679.783569:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767679.783573:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767679.783579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076807040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767679.783583:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350930a80 x1862023076806272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783599:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076806272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783605:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783609:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767679.783614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767679.783626:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350930000 x1862023076806400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076806400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783643:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767679.783651:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767679.783659:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783664:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350931880 x1862023076806528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783671:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076806528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783680:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076806528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767679.783683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767679.783692:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783698:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350930e00 x1862023076806656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783705:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076806656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783711:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783714:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767679.783725:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767679.783742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350932300 x1862023076806784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076806784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783782:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767679.783795:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767679.783815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783826:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350932a00 x1862023076806912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076806912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076806912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767679.783883:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767679.783897:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767679.783905:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767679/real 1775767679] req@ffffa04350930700 x1862023076807040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767679.783913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076807040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767679.783918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767679.783919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767679.783921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767679.975128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767679.975144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767680.807137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767680.807237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767680.999149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767680.999172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767681.831089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767681.831142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767682.023130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767682.023130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767682.855111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767682.855156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767683.047131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767683.047131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767683.879104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767683.879162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767684.071134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767684.071134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767684.903136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767684.903160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767684.903169:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903176:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767684.903179:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903187:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903195:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903201:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767684.903236:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076807168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903265:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903269:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767684.903272:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903294:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903299:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903308:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767684.903323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076807296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903333:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767684.903336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076807168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767684.903337:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767684.903340:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903348:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903352:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903355:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767684.903368:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076807424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903377:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903381:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767684.903384:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903388:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903449:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903455:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903460:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767684.903473:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076807552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767684.903497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903502:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903508:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903513:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903520:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767684.903533:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076807680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903542:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903544:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767684.903546:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903549:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903552:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903554:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903557:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767684.903565:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076807808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767684.903575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767684.903585:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903589:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767684.903591:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903597:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00080000:2.0:1775767684.903607:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903612:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903619:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767684.903633:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076807936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767684.903644:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767684.903648:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767684.903650:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903655:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00100000:3.0:1775767684.903662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7e680 x1862023076807168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767684.903671:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903678:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903689:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767684.903703:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076807168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767684.903706:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076808064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767684.903727:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767684.903730:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767684.903733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767684.903739:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767684.903740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076807168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767684.903742:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767684.903761:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4186 00000100:00100000:3.0:1775767684.903762:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807296:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767684.903769:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767684.903776:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767684.903782:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767684.903787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767684.903812:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076808192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767684.903817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7c380 x1862023076807296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.903831:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076807296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.903851:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.903855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.903858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767684.903863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767684.903880:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.903887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7dc00 x1862023076807424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.903894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076807424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.903899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.903901:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.903903:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767684.903907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767684.903918:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.903923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7c700 x1862023076807552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.903930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076807552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.903935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.903936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.903939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767684.903942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767684.903952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.903957:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7c000 x1862023076807680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.903963:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076807680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.903969:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.903970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.903972:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076807680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767684.903975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767684.903984:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.903989:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7f100 x1862023076807808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.903995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076807808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.904000:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.904002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.904004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767684.904007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767684.904016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.904022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7f480 x1862023076807936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.904052:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076807936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.904058:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.904061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.904063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076807936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767684.904067:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767684.904080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.904088:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7e300 x1862023076808064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.904100:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076808064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.904106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.904108:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.904110:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767684.904124:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767684.904134:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767684.904140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767684/real 1775767684] req@ffffa0434aa7e680 x1862023076808192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767684.904147:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076808192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767684.904152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767684.904154:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767684.904156:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767685.095095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767685.095112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767685.927126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767685.927138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767686.119089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767686.119093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767686.951132:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767686.951144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767687.143112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767687.143143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767687.975128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767687.975194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767688.167168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767688.167170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767688.999176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767688.999205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767689.191152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767689.191152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767690.023173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767690.023218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767690.023221:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.023231:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767690.023236:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023249:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023261:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767690.023269:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767690.023321:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076808320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767690.023355:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.023359:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767690.023361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023365:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023370:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767690.023396:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076808320:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767690.023400:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767690.023418:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767690.023440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076808448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767690.023453:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.023457:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767690.023459:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023463:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023476:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767690.023484:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767690.023490:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767690.023496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767690.023524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076808576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767690.023535:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.023539:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767690.023542:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767690.023548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350930700 x1862023076808320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.023549:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023558:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767690.023573:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767690.023580:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767690.023584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076808320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.023600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767690.023601:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076808704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767690.023617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767690.023627:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767690.023628:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076808320:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767690.023642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767690.023645:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023652:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023658:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767690.023666:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767690.023667:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808448:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767690.023681:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767690.023715:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076808832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767690.023723:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767690.023761:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.023766:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767690.023768:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350932a00 x1862023076808448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.023770:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023787:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023794:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767690.023796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076808448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.023810:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767690.023826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767690.023829:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767690.023833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.023840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767690.023851:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767690.023852:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076808960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767690.023879:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767690.023887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767690.023889:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767690.023894:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.023900:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00080000:1.0:1775767690.023906:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767690.023911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350932300 x1862023076808576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.023916:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767690.023931:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767690.023936:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076808576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.023952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767690.023957:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076809088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767690.023961:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.023971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767690.023993:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808704:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767690.024004:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.024009:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767690.024012:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767690.024019:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4191 00000800:00000100:3.0:1775767690.024059:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024073:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350930e00 x1862023076808704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.024074:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767690.024089:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024093:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076808704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767690.024103:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767690.024105:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024117:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767690.024128:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076809216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767690.024137:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808832:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767690.024151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767690.024157:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767690.024159:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767690.024164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767690.024178:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4191 00000100:00100000:3.0:1775767690.024184:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350931880 x1862023076808832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767690.024185:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767690.024196:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767690.024209:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767690.024220:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076808832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767690.024230:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076809344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767690.024242:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024249:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024253:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076808832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767690.024262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767690.024284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024292:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350930000 x1862023076808960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767690.024301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076808960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.024307:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024309:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024311:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076808960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767690.024315:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767690.024325:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024330:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350930a80 x1862023076809088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767690.024352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076809088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.024358:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024360:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767690.024373:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767690.024385:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024392:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350931c00 x1862023076809216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767690.024399:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076809216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.024404:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024407:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024409:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767690.024414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767690.024432:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767690.024442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767690/real 1775767690] req@ffffa04350932680 x1862023076809344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767690.024455:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076809344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767690.024471:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767690.024475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767690.024478:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076809344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767690.215101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767690.215128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767691.047128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767691.047206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767691.239120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767691.239120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767692.071152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767692.071163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767692.263098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767692.263110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767693.095089:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767693.095175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767693.287122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767693.287144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767694.119100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767694.119129:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767694.311132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767694.311133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767695.143128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767695.143152:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143161:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767695.143166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4196 00010000:00010000:3.0:1775767695.143182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767695.143188:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143197:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767695.143236:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076809472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767695.143269:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143274:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767695.143276:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143280:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00080000:1.0:1775767695.143285:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143289:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767695.143299:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767695.143315:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076809600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767695.143316:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076809472:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767695.143335:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143338:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767695.143340:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143359:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00080000:1.0:1775767695.143366:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143370:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767695.143376:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767695.143393:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076809728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767695.143393:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767695.143405:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767695.143411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00080000:1.0:1775767695.143420:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143425:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767695.143430:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350932680 x1862023076809472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767695.143431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767695.143450:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076809856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767695.143452:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076809472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.143483:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.143488:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.143493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076809472:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767695.143494:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767695.143504:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143509:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00080000:1.0:1775767695.143514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143517:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767695.143523:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767695.143525:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767695.143538:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076809984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767695.143554:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767695.143566:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767695.143568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350931c00 x1862023076809600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767695.143574:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767695.143579:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143588:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00100000:3.0:1775767695.143589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076809600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767695.143596:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143606:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767695.143607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767695.143613:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767695.143622:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.143626:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767695.143631:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076810112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767695.143635:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767695.143650:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143654:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767695.143656:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143661:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4196 00000800:00000100:3.0:1775767695.143667:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767695.143669:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767695.143675:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767695.143682:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767695.143686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350930a80 x1862023076809728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767695.143712:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076810240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767695.143723:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767695.143732:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767695.143736:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767695.143740:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00100000:3.0:1775767695.143743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076809728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.143755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767695.143757:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767695.143759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.143764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767695.143768:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767695.143773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809856:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767695.143777:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767695.143794:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076810368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767695.143799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767695.143810:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767695.143810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350930000 x1862023076809856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767695.143821:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767695.143825:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767695.143828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076809856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767695.143833:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4196 00000100:00080000:1.0:1775767695.143843:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767695.143843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.143850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767695.143852:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767695.143857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809856:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775767695.143861:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767695.143878:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076810496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767695.143879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767695.143910:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767695.143925:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350931880 x1862023076809984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767695.143936:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076809984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.143942:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.143944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.143946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076809984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767695.143951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767695.143965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767695.143971:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350930e00 x1862023076810112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767695.143979:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076810112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.143984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.143985:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.143987:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767695.143990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767695.144000:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767695.144006:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350932300 x1862023076810240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767695.144013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076810240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.144018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.144019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.144021:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767695.144043:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767695.144055:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767695.144062:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350932a00 x1862023076810368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767695.144069:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076810368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.144074:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.144076:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.144078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767695.144088:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767695.144098:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767695.144104:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767695/real 1775767695] req@ffffa04350930700 x1862023076810496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767695.144110:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076810496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767695.144116:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767695.144118:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767695.144120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076810496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767695.335125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767695.335125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767696.167120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767696.167207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767696.359140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767696.359161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767697.191148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767697.191188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767697.383108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767697.383122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767698.215130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767698.215131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767698.407149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767698.407154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767699.239188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767699.239322:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767699.431097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767699.431116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767700.263126:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775767700.263137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767700.263138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263149:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263158:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767700.263176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767700.263181:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767700.263211:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076810624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263232:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263241:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767700.263243:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263251:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263254:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263261:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767700.263272:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076810752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767700.263276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076810624:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767700.263283:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263287:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767700.263288:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263291:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263293:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263295:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263298:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767700.263305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076810880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263314:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767700.263319:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263322:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263327:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263329:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263334:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767700.263342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076811008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767700.263352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263355:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263356:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263359:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263363:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767700.263376:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076811136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263389:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767700.263389:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076810752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767700.263398:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767700.263400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263406:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263411:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263417:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767700.263424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076810880:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767700.263433:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767700.263442:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767700.263460:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076811264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767700.263466:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811136:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767700.263471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263474:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767700.263475:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263480:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263485:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767700.263490:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811264:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767700.263491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263499:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767700.263525:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076811392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263541:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263546:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767700.263548:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263553:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00080000:0.0:1775767700.263557:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767700.263564:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811392:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767700.263571:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263578:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767700.263592:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076811520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767700.263605:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767700.263609:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767700.263611:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767700.263622:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4201 00000100:00100000:3.0:1775767700.263629:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811520:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767700.263640:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767700.263671:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767700.263678:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767700.263692:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076811648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767700.263751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767700.455127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767700.455127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767701.287116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767701.287134:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767701.479114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767701.479115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767702.311141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767702.311184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767702.503132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767702.503132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767703.335101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767703.335123:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775767703.335284:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775767703.335320:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775767703.335336:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335386:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335401:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335410:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335418:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335424:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335432:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335439:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775767703.335446:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775767703.335495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349641500 x1862023076811648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076811648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767703.335552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349641180 x1862023076811520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076811520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335576:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767703.335597:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349643b80 x1862023076811392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335628:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076811392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335639:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335646:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767703.335660:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349643800 x1862023076811264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076811264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076811264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767703.335702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349641880 x1862023076811136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335713:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076811136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335726:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335729:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767703.335746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349640700 x1862023076810752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076810752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335777:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335781:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076810752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767703.335789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349641c00 x1862023076810880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076810880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076810880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767703.335829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349642680 x1862023076811008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335839:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076811008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335852:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335855:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767703.335870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767700/real 1775767703] req@ffffa04349640e00 x1862023076810624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767703.335885:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076810624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767703.335894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767703.335897:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767703.335901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076810624:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767703.527138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767703.527138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767704.359118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767704.359136:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767704.551141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767704.551141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767705.383141:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767705.383142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767705.383157:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767705.383162:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383174:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4207 00010000:00010000:3.0:1775767705.383178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767705.383188:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383199:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767705.383244:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076811776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383279:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383282:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767705.383283:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383287:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383289:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383293:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383326:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767705.383342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076811904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383354:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383357:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767705.383359:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383363:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383368:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383372:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767705.383388:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076812032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383399:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383404:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767705.383406:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383410:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383414:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383419:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767705.383441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076812160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767705.383455:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076811776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767705.383483:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383487:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767705.383502:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383508:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383519:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383527:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767705.383557:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076812288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383569:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383573:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767705.383575:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383580:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383585:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383591:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383607:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767705.383620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076812416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383629:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383632:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767705.383634:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383640:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383643:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383647:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383654:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767705.383661:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767705.383687:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076812544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767705.383700:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383704:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767705.383706:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383722:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00100000:3.0:1775767705.383732:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349640e00 x1862023076811776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767705.383736:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383755:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383771:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767705.383785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076811776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767705.383793:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076812672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767705.383852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767705.383853:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767705.383864:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767705.383870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767705.383872:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767705.383881:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4207 00000100:00080000:0.0:1775767705.383887:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767705.383892:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767705.383898:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767705.383912:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076812800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767705.383971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076811776:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767705.383981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767705.384012:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384052:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349642680 x1862023076811904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384068:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076811904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384078:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384081:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384085:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076811904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767705.384092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767705.384114:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384126:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349641c00 x1862023076812032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076812032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384147:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767705.384184:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767705.384209:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349640700 x1862023076812160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384233:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076812160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384246:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384250:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384254:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767705.384260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767705.384281:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384292:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349641880 x1862023076812288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384303:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076812288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384312:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384315:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384318:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076812288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767705.384325:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767705.384344:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384354:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349643800 x1862023076812416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384365:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076812416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384373:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384377:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384381:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767705.384388:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767705.384408:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349643b80 x1862023076812544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384432:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076812544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384440:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384444:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384447:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767705.384453:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767705.384476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349641180 x1862023076812672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384498:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076812672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384505:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384508:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384511:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767705.384518:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767705.384537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767705.384711:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767705/real 1775767705] req@ffffa04349641500 x1862023076812800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767705.384725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076812800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767705.384734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767705.384738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767705.384741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076812800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767705.575122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767705.575168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767706.407137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767706.407164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767706.599150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767706.599150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767707.431171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767707.431176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767707.623115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767707.623115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767708.455153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767708.455207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767708.647115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767708.647158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767709.479125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767709.479152:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767709.671132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767709.671132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767710.503183:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503199:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775767710.503205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767710.503205:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767710.503222:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4212 00010000:00010000:3.0:1775767710.503229:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767710.503238:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767710.503248:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767710.503299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076812928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767710.503332:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503336:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767710.503338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767710.503343:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:2.0:1775767710.503348:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767710.503353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076812928:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767710.503365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767710.503384:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076813056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767710.503396:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767710.503422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767710.503427:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:2.0:1775767710.503432:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767710.503439:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767710.503439:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767710.503449:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767710.503464:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076813184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767710.503477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503481:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767710.503484:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767710.503485:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7d500 x1862023076812928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503495:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:2.0:1775767710.503501:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767710.503507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076812928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503508:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767710.503518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767710.503520:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.503524:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767710.503527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076812928:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767710.503570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767710.503579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076813312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767710.503590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767710.503596:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503601:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767710.503604:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7fb80 x1862023076813056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503605:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767710.503616:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00100000:3.0:1775767710.503621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076813056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503622:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767710.503631:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767710.503634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767710.503639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767710.503639:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767710.503642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767710.503647:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767710.503667:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076813440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767710.503667:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7d880 x1862023076813184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503687:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503693:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767710.503693:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076813184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503698:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767710.503707:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767710.503708:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:3.0:1775767710.503716:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767710.503717:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767710.503719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767710.503726:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767710.503732:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767710.503732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767710.503751:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076813568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767710.503754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7df80 x1862023076813312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503769:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503775:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767710.503777:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076813312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503778:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767710.503787:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:2.0:1775767710.503794:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767710.503794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.503801:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767710.503802:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503807:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813312:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775767710.503809:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767710.503814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767710.503826:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076813696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767710.503831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7ea00 x1862023076813440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503843:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767710.503852:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767710.503855:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767710.503857:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076813440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767710.503863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.503869:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767710.503873:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00100000:3.0:1775767710.503873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076813440:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767710.503883:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767710.503888:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767710.503890:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767710.503899:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767710.503914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767710.503915:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076813824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767710.503924:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7ce00 x1862023076813568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503934:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767710.503939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076813568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767710.503941:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767710.503945:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767710.503954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767710.503955:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4212 00000100:00080000:3.0:1775767710.503960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767710.503963:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767710.503965:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767710.503974:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767710.503974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813696:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767710.503985:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767710.503993:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767710.504002:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076813952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767710.504002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7ed80 x1862023076813696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767710.504052:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076813696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767710.504066:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.504070:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767710.504073:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767710.504084:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767710.504097:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767710.504109:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7f800 x1862023076813824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767710.504117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076813824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767710.504122:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.504124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767710.504126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767710.504130:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767710.504139:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767710.504145:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767710/real 1775767710] req@ffffa0434aa7ea00 x1862023076813952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767710.504153:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076813952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767710.504158:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767710.504160:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767710.504161:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076813952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767710.695115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767710.695146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767711.527119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767711.527134:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767711.719094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767711.719095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767712.551124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767712.551184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767712.743146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767712.743146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767713.575137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767713.575148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767713.767138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767713.767147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767714.599146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767714.599181:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767714.791103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767714.791154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767715.623125:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767715.623128:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623136:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767715.623140:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623160:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623179:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623188:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767715.623210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775767715.623233:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076814080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767715.623272:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623275:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767715.623276:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623280:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623284:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623288:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767715.623296:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767715.623309:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076814208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767715.623315:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076814080:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767715.623318:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623325:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767715.623327:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623330:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623333:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623335:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767715.623338:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767715.623347:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076814336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767715.623353:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623356:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767715.623358:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623360:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623376:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623380:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767715.623385:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767715.623389:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767715.623399:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076814464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767715.623406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767715.623411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623419:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623424:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623427:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7df80 x1862023076814080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767715.623430:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767715.623444:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076814592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767715.623451:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076814080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623466:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623473:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767715.623477:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767715.623477:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076814080:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767715.623484:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767715.623487:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623492:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623508:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623512:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814208:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767715.623518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767715.623533:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076814720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767715.623550:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623564:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7d880 x1862023076814208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767715.623580:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623583:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767715.623585:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076814208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767715.623586:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767715.623594:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623601:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767715.623602:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767715.623607:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767715.623610:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767715.623613:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767715.623614:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767715.623623:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767715.623628:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076814848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767715.623636:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767715.623639:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767715.623642:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767715.623642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767715.623649:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00080000:2.0:1775767715.623653:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767715.623656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7fb80 x1862023076814336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767715.623659:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767715.623666:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767715.623673:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076814336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767715.623678:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076814976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767715.623710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767715.623711:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767715.623717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767715.623718:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767715.623722:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767715.623725:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767715.623729:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4217 00000100:00100000:3.0:1775767715.623733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814464:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767715.623738:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767715.623744:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767715.623749:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767715.623754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767715.623762:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076815104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767715.623766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7d500 x1862023076814464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.623790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076814464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.623802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767715.623807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767715.623826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7e680 x1862023076814592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.623839:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076814592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623845:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.623849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076814592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767715.623860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767715.623871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7e300 x1862023076814720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.623884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076814720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.623894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767715.623897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767715.623907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7f480 x1862023076814848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.623922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076814848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.623936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767715.623942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767715.623971:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.623982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7f100 x1862023076814976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.623992:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076814976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.623997:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.623999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.624001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076814976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767715.624005:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076815104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767715.624018:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767715.624050:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767715/real 1775767715] req@ffffa0434aa7c000 x1862023076815104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767715.624057:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076815104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767715.624063:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767715.624065:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767715.624067:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076815104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767715.815140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767715.815177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767716.647174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767716.647193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767716.839114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767716.839144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767717.671127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767717.671148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767717.863154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767717.863179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767718.695101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767718.695157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767718.887127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767718.887127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767719.719119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767719.719145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767719.911103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767719.911128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767720.743094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767720.743106:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743112:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767720.743116:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743125:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743133:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743139:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767720.743172:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076815232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743204:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767720.743207:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743212:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743217:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743224:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743225:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076815232:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767720.743236:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767720.743255:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076815360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743272:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1775767720.743273:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1775767720.743284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767720.743290:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767720.743292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743299:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743305:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743311:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743318:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767720.743334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076815488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743349:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767720.743352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743356:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743359:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743363:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743368:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767720.743378:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076815616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743386:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767720.743389:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7c700 x1862023076815232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767720.743390:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767720.743393:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743400:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743404:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743410:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743420:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076815232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767720.743421:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767720.743432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767720.743439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076815744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767720.743442:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076815232:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767720.743471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743474:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767720.743477:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743484:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743489:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743494:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743499:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767720.743511:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076815872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743519:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743521:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767720.743523:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743525:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743528:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743530:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743533:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767720.743540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076816000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767720.743548:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767720.743554:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743556:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743559:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743562:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767720.743575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076816128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767720.743586:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767720.743586:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767720.743595:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767720.743598:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767720.743603:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4222 00000100:00080000:2.0:1775767720.743608:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767720.743613:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767720.743619:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767720.743620:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767720.743638:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076816256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767720.743685:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7dc00 x1862023076815360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076815360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743707:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743710:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767720.743718:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767720.743733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7c380 x1862023076815488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743749:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076815488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743754:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743756:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743759:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767720.743762:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767720.743776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743786:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7c000 x1862023076815616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076815616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743820:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767720.743826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767720.743845:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7f100 x1862023076815744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076815744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076815744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767720.743883:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076815872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767720.743899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7f480 x1862023076815872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743921:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076815872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743934:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076815872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767720.743944:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767720.743964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.743973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7e300 x1862023076816000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.743981:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076816000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.743986:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.743988:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.743991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767720.743995:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767720.744007:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.744017:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7e680 x1862023076816128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.744059:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076816128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.744068:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.744071:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.744074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767720.744081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767720.744103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767720.744114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767720/real 1775767720] req@ffffa0434aa7d500 x1862023076816256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767720.744127:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076816256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767720.744136:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767720.744139:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767720.744142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076816256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767720.935129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767720.935174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767721.767154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767721.767171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767721.959170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767721.959185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767722.791148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767722.791198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767722.983174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767722.983174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767723.815165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767723.815195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767724.007143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767724.007143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767724.839124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767724.839170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767725.031129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767725.031132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767725.863180:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863190:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775767725.863207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767725.863216:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767725.863218:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767725.863227:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863237:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863243:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767725.863275:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076816384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767725.863298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863301:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767725.863302:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863305:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863308:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863311:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863319:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767725.863328:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076816512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767725.863332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076816384:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767725.863343:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863345:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767725.863346:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863349:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863351:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767725.863363:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076816640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767725.863379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863382:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767725.863384:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863386:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863390:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863393:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863413:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767725.863422:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767725.863427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076816768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767725.863438:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863442:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767725.863444:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863448:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863453:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863459:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863460:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349640a80 x1862023076816384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767725.863467:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767725.863481:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076816384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767725.863483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076816896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767725.863506:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767725.863507:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863512:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767725.863515:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767725.863516:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767725.863521:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076816384:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767725.863522:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863531:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863535:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863539:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767725.863542:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767725.863549:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076817024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767725.863558:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775767725.863564:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767725.863566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863572:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00100000:3.0:1775767725.863576:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349642300 x1862023076816512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767725.863578:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863588:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863598:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767725.863602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076816512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863615:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767725.863616:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076817152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767725.863619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767725.863628:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816640:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767725.863638:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863641:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767725.863642:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863646:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4227 00000800:00000100:3.0:1775767725.863650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767725.863653:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863658:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349642a00 x1862023076816640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767725.863664:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767725.863674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076816640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767725.863680:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076817280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767725.863682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863694:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767725.863700:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816768:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767725.863713:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767725.863716:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767725.863717:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767725.863721:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4227 00000100:00080000:0.0:1775767725.863724:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767725.863726:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767725.863730:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767725.863732:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767725.863740:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349640380 x1862023076816768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767725.863745:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076817408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767725.863763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076816768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863774:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863777:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767725.863783:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767725.863795:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349643100 x1862023076816896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767725.863808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076816896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076816896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767725.863830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767725.863840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349640a80 x1862023076817024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767725.863853:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076817024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767725.863866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767725.863875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863880:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349641500 x1862023076817152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767725.863886:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076817152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767725.863898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767725.863906:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349641180 x1862023076817280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767725.863917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076817280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767725.863928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767725.863937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767725.863942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767725/real 1775767725] req@ffffa04349643b80 x1862023076817408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767725.863949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076817408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767725.863955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767725.863956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767725.863958:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076817408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767726.055162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767726.055162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767726.887171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767726.887252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767727.079126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767727.079173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767727.911193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767727.911222:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767728.103150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767728.103151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767728.935128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767728.935212:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767729.127099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767729.127100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767729.959143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767729.959165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767730.151131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767730.151131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767730.983149:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983161:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767730.983161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767730.983165:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4232 00010000:00010000:3.0:1775767730.983181:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767730.983188:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983195:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767730.983229:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076817536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983272:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983277:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767730.983279:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983283:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00080000:0.0:1775767730.983288:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983291:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983301:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076817536:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767730.983308:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767730.983323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076817664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983331:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983334:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767730.983336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983338:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00080000:0.0:1775767730.983357:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983360:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767730.983364:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767730.983370:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767730.983376:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076817792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983387:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767730.983388:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983391:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00080000:0.0:1775767730.983400:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983404:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767730.983408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767730.983409:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349643800 x1862023076817536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767730.983420:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076817920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767730.983427:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076817536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767730.983429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983438:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767730.983442:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767730.983445:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767730.983449:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00080000:3.0:1775767730.983453:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767730.983456:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767730.983458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076817536:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767730.983464:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767730.983470:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767730.983480:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767730.983485:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076818048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983498:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767730.983502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767730.983505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767730.983509:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983516:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00100000:3.0:1775767730.983516:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349641880 x1862023076817664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767730.983523:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983530:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767730.983538:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767730.983543:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076817664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767730.983550:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076818176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767730.983551:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983560:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983567:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767730.983580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767730.983581:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767730.983586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767730.983588:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983592:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00080000:0.0:1775767730.983595:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983597:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767730.983601:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767730.983602:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767730.983611:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349640700 x1862023076817792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767730.983614:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076818304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983637:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767730.983639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076817792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767730.983641:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767730.983645:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767730.983654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767730.983659:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00100000:3.0:1775767730.983663:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817792:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767730.983667:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983674:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817920:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767730.983682:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767730.983696:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076818432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767730.983708:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767730.983708:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767730.983714:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767730.983717:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767730.983723:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4232 00000100:00100000:3.0:1775767730.983725:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349641c00 x1862023076817920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767730.983731:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767730.983737:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076817920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767730.983744:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767730.983757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767730.983759:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076818560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767730.983763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076817920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767730.983792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767730.983809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349642680 x1862023076818048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767730.983825:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076818048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767730.983831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983835:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767730.983839:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767730.983848:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983854:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349640e00 x1862023076818176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767730.983861:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076818176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767730.983866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767730.983874:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767730.983883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349643800 x1862023076818304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767730.983894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076818304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767730.983899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983901:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983903:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767730.983907:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767730.983922:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349643b80 x1862023076818432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767730.983943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076818432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767730.983952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.983955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.983959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767730.983965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767730.983983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767730.983993:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767730/real 1775767730] req@ffffa04349641180 x1862023076818560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767730.984005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076818560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767730.984012:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767730.984016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767730.984019:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076818560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767731.175109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767731.175117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767732.007131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767732.007164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767732.199104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767732.199124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767733.031122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767733.031133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767733.223153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767733.223158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767734.055123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767734.055163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767734.247143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767734.247146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767735.079093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767735.079146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767735.271132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767735.271132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767736.103125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767736.103169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767736.103277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767736.103287:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103308:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103324:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103332:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767736.103371:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076818688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767736.103404:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767736.103408:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103412:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103415:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103419:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767736.103428:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767736.103437:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076818688:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767736.103447:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076818816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767736.103456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103459:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767736.103460:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103470:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103474:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103477:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767736.103481:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767736.103491:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076818944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767736.103498:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767736.103504:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103507:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767736.103509:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103512:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103517:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103520:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767736.103524:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767736.103532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076819072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767736.103537:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350930700 x1862023076818688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767736.103555:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767736.103556:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076818688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767736.103563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767736.103567:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103574:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:3.0:1775767736.103574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767736.103580:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767736.103582:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767736.103586:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103590:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076818688:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767736.103597:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767736.103607:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076819200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767736.103610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767736.103625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767736.103630:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103634:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103638:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767736.103643:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767736.103645:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767736.103650:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767736.103660:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076819328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767736.103665:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350932a00 x1862023076818816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767736.103670:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767736.103681:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103689:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00080000:1.0:1775767736.103692:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103694:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103697:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076818816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767736.103699:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767736.103710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767736.103711:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076819456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767736.103717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.103724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767736.103732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767736.103735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767736.103736:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767736.103747:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103753:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4237 00000800:00000100:3.0:1775767736.103756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350932300 x1862023076818944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767736.103766:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103776:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103778:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076818944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767736.103785:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767736.103786:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.103792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.103795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076818944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767736.103800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767736.103804:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076819584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767736.103812:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767736.103816:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775767736.103817:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767736.103819:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767736.103824:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4237 00000100:00100000:3.0:1775767736.103828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350930e00 x1862023076819072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767736.103830:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767736.103838:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076819072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767736.103843:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767736.103857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767736.103868:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076819712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767736.103869:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.103876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767736.103898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767736.103914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350931880 x1862023076819200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767736.103928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076819200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767736.103934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.103936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.103938:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767736.103942:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767736.103952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.103957:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350930000 x1862023076819328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767736.103972:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076819328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767736.103978:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.103980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.103982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767736.103986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767736.103997:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.104003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350930a80 x1862023076819456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767736.104010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076819456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767736.104015:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.104016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.104018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767736.104022:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767736.104050:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.104057:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350931c00 x1862023076819584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767736.104065:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076819584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767736.104070:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.104072:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.104074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767736.104077:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767736.104087:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767736.104092:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767736/real 1775767736] req@ffffa04350932680 x1862023076819712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767736.104099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076819712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767736.104105:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767736.104106:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767736.104108:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076819712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767736.295112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767736.295115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767737.127106:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767737.127121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767737.319163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767737.319273:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767738.151116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767738.151138:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767738.343094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767738.343114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767739.175129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767739.175160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767739.367113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767739.367113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767740.199136:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767740.199143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767740.391123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767740.391123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767741.223112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767741.223114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767741.223128:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223138:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223149:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223156:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767741.223161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767741.223204:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076819840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767741.223227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767741.223231:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223234:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223237:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223240:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767741.223275:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076819968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767741.223281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076819840:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767741.223286:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223289:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767741.223290:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223293:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223296:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223298:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767741.223309:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076820096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767741.223316:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767741.223319:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223327:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223331:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223334:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767741.223341:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767741.223349:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076820224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767741.223357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767741.223360:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223363:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223365:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223368:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223371:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767741.223376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350932680 x1862023076819840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767741.223381:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076820352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767741.223399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767741.223403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767741.223407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076819840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767741.223410:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223415:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223421:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223425:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767741.223425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223432:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767741.223435:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076820480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767741.223439:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076819840:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767741.223449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223451:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767741.223453:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223456:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223459:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767741.223460:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767741.223465:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223469:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767741.223480:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076820608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767741.223487:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767741.223495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223498:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767741.223500:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767741.223503:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350931c00 x1862023076819968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767741.223504:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00080000:1.0:1775767741.223509:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223514:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223520:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767741.223525:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076819968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767741.223530:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076820736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767741.223545:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767741.223548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767741.223552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767741.223552:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767741.223554:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767741.223559:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4242 00000100:00100000:3.0:1775767741.223559:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076819968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767741.223566:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767741.223567:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767741.223573:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767741.223578:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767741.223587:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767741.223590:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076820864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767741.223597:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350930a80 x1862023076820096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223607:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076820096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223613:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223615:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223618:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767741.223621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767741.223633:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350930000 x1862023076820224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223645:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076820224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223653:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223655:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767741.223658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767741.223667:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223672:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350931880 x1862023076820352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076820352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223688:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076820352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767741.223691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767741.223700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350930e00 x1862023076820480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223712:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076820480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223717:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223719:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767741.223733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767741.223743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223748:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350932300 x1862023076820608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223755:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076820608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223760:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767741.223768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767741.223776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350932a00 x1862023076820736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223788:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076820736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223797:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767741.223800:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767741.223808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767741.223814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767741/real 1775767741] req@ffffa04350930700 x1862023076820864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767741.223821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076820864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767741.223826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767741.223827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767741.223829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076820864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767741.415127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767741.415127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767742.247149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767742.247176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767742.439141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767742.439144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767743.271126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767743.271158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767743.463126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767743.463126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767744.295120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767744.295168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767744.487100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767744.487109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767745.319170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767745.319170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767745.511122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767745.511122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767746.343092:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767746.343108:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343119:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343131:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767746.343138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767746.343139:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767746.343180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767746.343185:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076820992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767746.343230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767746.343238:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343244:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00100000:2.0:1775767746.343258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076820992:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767746.343265:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767746.343270:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343279:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767746.343290:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076821120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767746.343299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767746.343303:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343306:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343309:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767746.343311:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343314:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767746.343321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076821248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767746.343328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767746.343335:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767746.343335:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767746.343342:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343344:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767746.343348:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343354:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767746.343364:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076821376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767746.343381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350930700 x1862023076820992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767746.343399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767746.343407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076820992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767746.343408:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343417:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343423:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767746.343425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767746.343429:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767746.343433:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767746.343435:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767746.343438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076820992:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767746.343448:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076821504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767746.343455:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343457:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767746.343458:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767746.343461:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767746.343464:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343479:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767746.343485:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767746.343490:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343492:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767746.343503:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076821632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767746.343508:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350932a00 x1862023076821120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767746.343513:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767746.343524:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343529:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343537:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767746.343539:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076821120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767746.343543:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343549:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767746.343554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343560:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767746.343562:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076821760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767746.343566:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767746.343578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767746.343582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767746.343589:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767746.343594:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343599:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767746.343603:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767746.343607:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767746.343614:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767746.343623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350932300 x1862023076821248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767746.343628:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076821888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767746.343662:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767746.343666:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767746.343669:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767746.343669:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076821248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767746.343678:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4248 00000100:00080000:1.0:1775767746.343688:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767746.343690:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767746.343695:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767746.343699:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767746.343702:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767746.343707:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767746.343716:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076822016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767746.343719:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767746.343750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.343762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350930e00 x1862023076821376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.343771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076821376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.343777:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343779:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.343782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767746.343786:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767746.343796:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.343802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350931880 x1862023076821504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.343809:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076821504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.343824:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.343828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076821504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767746.343843:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767746.343863:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.343874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350930000 x1862023076821632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.343887:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076821632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.343897:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343900:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.343903:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767746.343909:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767746.343924:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.343931:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350930a80 x1862023076821760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.343938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076821760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.343944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.343953:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767746.343957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767746.343969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.343975:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350931c00 x1862023076821888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.343982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076821888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.343987:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.343989:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.343990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076821888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767746.343994:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767746.344003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767746.344008:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767746/real 1775767746] req@ffffa04350932680 x1862023076822016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767746.344015:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076822016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767746.344019:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767746.344021:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767746.344023:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767746.535142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767746.535142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767747.367119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767747.367137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767747.559129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767747.559140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767748.391110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767748.391195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767748.583120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767748.583121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767749.415129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767749.415205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767749.607124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767749.607124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767750.439105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767750.439172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767750.631076:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767750.631116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767751.463096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767751.463138:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463145:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767751.463162:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463172:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4253 00010000:00010000:2.0:1775767751.463177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767751.463183:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463190:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767751.463226:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076822144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463248:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463251:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767751.463253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463256:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463259:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463263:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463271:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767751.463283:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076822272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767751.463285:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076822144:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767751.463298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463300:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767751.463301:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463304:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463306:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463308:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463311:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767751.463318:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076822400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463324:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463326:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767751.463328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463331:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463341:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463344:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463347:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767751.463355:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767751.463356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076822528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767751.463368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463371:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463374:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463379:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767751.463386:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076822656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463391:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463394:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767751.463395:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463398:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463401:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767751.463403:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825ce00 x1862023076822144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767751.463406:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767751.463421:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076822784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463428:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463431:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767751.463433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767751.463435:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076822144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767751.463438:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463441:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463446:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463453:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767751.463462:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767751.463464:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076822912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767751.463471:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767751.463473:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767751.463477:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767751.463479:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767751.463480:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076822144:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767751.463484:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463489:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463492:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463495:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767751.463501:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076823040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767751.463508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767751.463513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822272:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767751.463513:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767751.463515:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767751.463522:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4253 00000100:00080000:3.0:1775767751.463524:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767751.463527:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767751.463530:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767751.463537:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076823168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767751.463547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463564:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825d880 x1862023076822272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076822272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463588:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463595:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767751.463601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767751.463635:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463648:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825dc00 x1862023076822400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076822400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767751.463697:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767751.463719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825d500 x1862023076822528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076822528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463755:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463758:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767751.463764:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767751.463784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463795:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825e300 x1862023076822656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076822656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463845:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076822656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767751.463860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767751.463883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463895:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825f100 x1862023076822784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076822784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767751.463929:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767751.463949:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.463960:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825e680 x1862023076822912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.463973:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076822912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.463981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.463984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.463994:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076822912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767751.464014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767751.464066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.464078:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825c000 x1862023076823040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.464090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076823040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.464099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.464102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.464106:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767751.464112:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767751.464132:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767751.464142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767751/real 1775767751] req@ffffa0435825ed80 x1862023076823168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767751.464170:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076823168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767751.464179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767751.464182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767751.464186:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767751.655107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767751.655108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767752.487107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767752.487188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767752.679093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767752.679119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767753.511105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767753.511163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767753.703123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767753.703123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767754.535109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767754.535180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767754.727148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767754.727149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767755.559145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767755.559287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767755.751125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767755.751125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767756.583152:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767756.583158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767756.583160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767756.583166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583175:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4258 00010000:00010000:3.0:1775767756.583176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767756.583184:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583191:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767756.583223:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076823296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767756.583243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767756.583246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767756.583248:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583251:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583254:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583257:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583284:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767756.583285:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076823296:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767756.583299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076823424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767756.583308:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767756.583311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767756.583312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583315:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583321:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583324:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767756.583332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076823552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767756.583339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767756.583343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767756.583344:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767756.583347:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583350:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583352:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583355:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767756.583367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076823680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767756.583374:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767756.583377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767756.583378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767756.583381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349641500 x1862023076823296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583391:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583395:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767756.583398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076823296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583401:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583407:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775767756.583414:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767756.583419:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076823808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767756.583419:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583426:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076823296:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767756.583436:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767756.583438:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767756.583439:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583443:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583445:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583448:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767756.583452:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767756.583462:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076823936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767756.583470:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767756.583474:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1775767756.583475:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767756.583476:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583481:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767756.583490:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349640a80 x1862023076823424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767756.583509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076824064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767756.583509:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076823424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583537:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767756.583538:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583544:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767756.583545:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1775767756.583547:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767756.583549:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767756.583554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823552:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767756.583556:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583562:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767756.583566:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583571:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767756.583589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076824192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767756.583598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767756.583606:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767756.583609:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349643100 x1862023076823552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583610:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767756.583613:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767756.583621:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4258 00000100:00080000:0.0:1775767756.583626:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767756.583628:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076823552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767756.583632:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767756.583640:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767756.583641:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583647:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767756.583654:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076824320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767756.583657:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767756.583677:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583685:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349640380 x1862023076823680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583693:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076823680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767756.583707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767756.583718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349642a00 x1862023076823808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076823808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583752:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076823808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767756.583770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767756.583787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583795:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349642300 x1862023076823936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076823936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076823936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767756.583816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767756.583827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349641500 x1862023076824064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076824064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583856:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767756.583869:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767756.583889:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349641180 x1862023076824192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076824192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583913:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767756.583920:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767756.583934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767756.583940:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767756/real 1775767756] req@ffffa04349643b80 x1862023076824320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767756.583949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076824320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767756.583953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767756.583955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767756.583964:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076824320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767756.775150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767756.775150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767757.607158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767757.607224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767757.799123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767757.799125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767758.631134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767758.631148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767758.823149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767758.823149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767759.655134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767759.655192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767759.847148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767759.847148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767760.679149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767760.679187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767760.871184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767760.871184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767761.703137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767761.703168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767761.703185:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703198:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703212:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703223:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767761.703232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767761.703274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076824448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703308:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703311:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767761.703313:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703317:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703321:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703326:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703337:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767761.703344:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076824448:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767761.703355:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076824576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703365:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703374:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767761.703377:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703381:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703383:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703386:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703393:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767761.703406:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076824704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767761.703415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767761.703428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767761.703434:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703440:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703446:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703452:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767761.703459:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349643800 x1862023076824448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767761.703460:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767761.703488:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076824832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703503:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703509:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767761.703509:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076824448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767761.703516:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703526:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703534:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703545:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767761.703547:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767761.703551:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:2.0:1775767761.703556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.703562:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076824448:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767761.703570:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076824960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703577:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703580:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767761.703581:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703587:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00100000:2.0:1775767761.703592:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824576:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767761.703594:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703605:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703612:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767761.703621:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076825088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703631:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703634:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767761.703635:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703640:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703648:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767761.703648:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767761.703661:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703667:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767761.703681:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076825216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767761.703684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349640e00 x1862023076824576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767761.703706:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703708:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767761.703713:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767761.703714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076824576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767761.703718:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:0.0:1775767761.703726:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767761.703733:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703745:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767761.703754:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767761.703756:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076825344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767761.703763:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767761.703766:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767761.703767:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767761.703780:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4263 00000100:00080000:2.0:1775767761.703783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767761.703784:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767761.703789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824576:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767761.703790:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767761.703803:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767761.703807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767761.703813:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076825472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767761.703835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.703846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349642680 x1862023076824704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.703855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076824704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.703861:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.703864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.703866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767761.703870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767761.703882:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.703888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349641c00 x1862023076824832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.703895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076824832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.703900:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.703902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.703904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767761.703908:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767761.703926:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.703933:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349640700 x1862023076824960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.703943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076824960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.703952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.703955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.703959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076824960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767761.703993:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767761.704016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.704071:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349641880 x1862023076825088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.704086:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076825088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.704093:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.704097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.704100:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767761.704107:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767761.704129:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.704140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349643800 x1862023076825216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.704168:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076825216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.704187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.704195:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.704199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767761.704204:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767761.704222:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.704230:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349643b80 x1862023076825344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.704238:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076825344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.704243:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.704245:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.704247:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767761.704250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767761.704260:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767761.704266:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767761/real 1775767761] req@ffffa04349641180 x1862023076825472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767761.704272:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076825472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767761.704287:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767761.704289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767761.704291:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076825472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767761.895205:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767761.895205:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767762.727199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767762.727301:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767762.919201:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767762.919220:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767763.751173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767763.751183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767763.943116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767763.943144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767764.775126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767764.775195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767764.967139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767764.967140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767765.799127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767765.799150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767765.991108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767765.991111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767766.823136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767766.823140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767766.823149:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767766.823155:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823167:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823177:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823183:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767766.823222:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076825600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823244:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823249:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767766.823260:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823284:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767766.823293:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076825600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767766.823300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076825728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823314:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823318:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767766.823320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823324:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823329:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823332:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823337:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767766.823352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076825856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823369:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823373:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767766.823376:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823381:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00100000:2.0:1775767766.823383:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767766.823387:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823393:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823400:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767766.823412:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767766.823415:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076825984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823433:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767766.823436:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767766.823436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825984:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767766.823443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823447:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823452:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823459:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767766.823474:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076826112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775767766.823475:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767766.823490:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767766.823496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767766.823503:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076826112:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767766.823504:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823512:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767766.823518:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767766.823539:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076826240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823555:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823559:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767766.823561:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823565:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823571:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767766.823574:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826240:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767766.823579:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767766.823601:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076826368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823617:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823621:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767766.823623:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823627:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823632:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767766.823635:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767766.823640:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823648:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767766.823661:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076826496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767766.823680:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767766.823683:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767766.823685:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767766.823689:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4268 00000100:00080000:0.0:1775767766.823693:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767766.823696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826496:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767766.823700:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767766.823708:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767766.823722:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076826624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767766.823740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767767.015171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767767.015171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767767.847098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767767.847166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767768.039108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767768.039111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767768.871121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767768.871234:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767769.063120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767769.063121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767769.895136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767769.895298:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775767769.895333:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775767769.895375:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767769.895390:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767769.895450:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775767769.895464:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767769.895475:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767769.895491:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349643800 x1862023076826624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767769.895506:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775767769.895517:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767769.895524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076826624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767769.895538:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:2.0:1775767769.895553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1775767769.895557:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:2.0:1775767769.895568:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000800:00000100:1.0:1775767769.895575:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775767769.895576:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767769.895589:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349643b80 x1862023076826496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076826496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895618:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767769.895631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349641180 x1862023076826368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895643:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076826368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895663:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767769.895674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349640a80 x1862023076826240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076826240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076826240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767769.895707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349643100 x1862023076826112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076826112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895727:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895730:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895734:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076826112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767769.895739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349640380 x1862023076825984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076825984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895778:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895781:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767769.895789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349642300 x1862023076825728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076825728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895820:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767769.895826:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349642a00 x1862023076825856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076825856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076825856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767769.895860:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767766/real 1775767769] req@ffffa04349641500 x1862023076825600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767769.895874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076825600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767769.895881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767769.895891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767769.895895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076825600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767770.087109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767770.087114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767770.919166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767770.919241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767771.111151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767771.111151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767771.943156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767771.943185:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943195:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767771.943199:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943211:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943224:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943232:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767771.943271:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076826752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767771.943303:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767771.943310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943314:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943320:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943326:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943340:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767771.943353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076826752:192.168.202.158@tcp:250:kworker.0 00010000:00010000:0.0:1775767771.943361:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767771.943362:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076826880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767771.943379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767771.943386:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943391:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943412:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943417:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943424:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767771.943442:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076827008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767771.943453:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943456:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767771.943458:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943462:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943466:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943475:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767771.943476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943487:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767771.943504:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076827136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767771.943515:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943519:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767771.943521:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943528:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943534:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943542:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767771.943549:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7fb80 x1862023076826752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767771.943553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767771.943581:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076827264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767771.943592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076826752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.943626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.943634:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.943650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076826752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767771.943664:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943669:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767771.943672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943680:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943687:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767771.943693:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076826880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767771.943694:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943711:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767771.943731:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076827392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767771.943744:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767771.943762:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943767:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767771.943769:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767771.943774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7d880 x1862023076826880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767771.943781:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943793:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943802:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943812:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767771.943814:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076826880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.943833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767771.943838:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076827520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767771.943844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.943856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076826880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767771.943876:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767771.943877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827008:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767771.943894:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767771.943898:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943903:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943915:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943920:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943926:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767771.943941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076827648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767771.943962:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767771.943966:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767771.943968:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767771.943972:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4273 00000100:00080000:2.0:1775767771.943976:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767771.943979:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767771.943984:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767771.943998:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076827776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767771.944080:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944104:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7df80 x1862023076827008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076827008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944133:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944139:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767771.944151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767771.944175:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944187:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7ea00 x1862023076827136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944199:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076827136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944207:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944211:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944214:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767771.944221:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767771.944245:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944257:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7f800 x1862023076827264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944275:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076827264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944290:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944293:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076827264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767771.944316:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767771.944340:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7ed80 x1862023076827392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944366:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076827392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944376:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944379:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944383:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767771.944391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767771.944411:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944421:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7ce00 x1862023076827520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944433:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076827520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944449:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944456:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767771.944463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767771.944482:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7fb80 x1862023076827648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944506:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076827648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944514:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944518:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944521:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767771.944527:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767771.944546:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767771.944555:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767771/real 1775767771] req@ffffa0434aa7d500 x1862023076827776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767771.944568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076827776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767771.944577:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767771.944580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767771.944584:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076827776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767772.135154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767772.135161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767772.967177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767772.967405:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767773.159115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767773.159141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767773.991123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767773.991210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767774.183186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767774.183186:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767775.015157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767775.015222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767775.207142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767775.207149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767776.039126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767776.039235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767776.231166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767776.231166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767777.063148:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063159:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767777.063165:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063178:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00080000:2.0:1775767777.063191:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063200:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767777.063209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767777.063217:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767777.063259:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076827904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767777.063307:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063310:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767777.063312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063316:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00080000:2.0:1775767777.063319:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063323:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063335:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767777.063354:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076828032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767777.063360:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076827904:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767777.063383:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767777.063388:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063392:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00080000:2.0:1775767777.063397:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063399:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063404:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767777.063414:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076828160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767777.063423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063427:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767777.063428:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063431:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4278 00000800:00000100:3.0:1775767777.063435:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767777.063447:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063453:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063459:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767777.063471:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076828288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767777.063478:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063481:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767777.063483:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063493:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00100000:3.0:1775767777.063496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7e680 x1862023076827904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767777.063497:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063502:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063508:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767777.063521:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076828416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767777.063530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767777.063535:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063539:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00100000:3.0:1775767777.063545:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076827904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767777.063546:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063556:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063570:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767777.063586:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767777.063594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076828544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767777.063601:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.063605:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076827904:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767777.063633:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767777.063641:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063647:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767777.063650:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063659:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4278 00000800:00000100:3.0:1775767777.063660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767777.063672:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063681:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767777.063681:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7e300 x1862023076828032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767777.063695:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767777.063700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076828032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.063709:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.063715:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767777.063720:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076828672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767777.063721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767777.063738:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767777.063759:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063763:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767777.063765:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063771:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00080000:2.0:1775767777.063777:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063780:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767777.063785:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767777.063789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767777.063807:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076828800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767777.063811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7f480 x1862023076828160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.063831:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076828160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767777.063842:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767777.063854:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767777.063857:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767777.063865:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4278 00000100:00080000:3.0:1775767777.063871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767777.063874:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767777.063881:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767777.063881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767777.063889:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767777.063917:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076828928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767777.063922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767777.063950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767777.063976:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.063986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7f100 x1862023076828288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.063994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076828288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.063999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767777.064008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767777.064019:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.064024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7c000 x1862023076828416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.064057:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076828416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.064064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064066:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064068:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076828416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767777.064083:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767777.064096:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.064103:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7c380 x1862023076828544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.064116:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076828544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.064124:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064127:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767777.064137:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767777.064157:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.064168:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7dc00 x1862023076828672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.064178:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076828672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.064188:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064190:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064192:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767777.064197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767777.064211:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.064217:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7c700 x1862023076828800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.064224:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076828800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.064230:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064231:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064233:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767777.064237:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767777.064246:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767777.064252:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767777/real 1775767777] req@ffffa0434aa7e680 x1862023076828928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767777.064260:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076828928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767777.064269:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767777.064272:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767777.064276:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076828928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767777.255102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767777.255102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767778.087148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767778.087250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767778.279129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767778.279129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767779.111151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767779.111189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767779.303156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767779.303281:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767780.135126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767780.135165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767780.327128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767780.327128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767781.159143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767781.159195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767781.351135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767781.351135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767782.183128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767782.183145:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767782.183155:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4283 00010000:00010000:3.0:1775767782.183156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767782.183167:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183175:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767782.183181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775767782.183215:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076829056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767782.183242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767782.183248:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767782.183253:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00080000:0.0:1775767782.183266:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767782.183269:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076829056:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767782.183280:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183292:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767782.183308:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076829184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767782.183321:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183324:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767782.183327:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767782.183331:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767782.183336:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00080000:0.0:1775767782.183343:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183347:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183353:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767782.183368:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076829312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183370:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349641880 x1862023076829056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767782.183408:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076829056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183412:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767782.183416:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767782.183423:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767782.183425:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00080000:3.0:1775767782.183430:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767782.183432:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767782.183433:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076829056:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767782.183443:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183449:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767782.183458:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767782.183465:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076829440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767782.183478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183482:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767782.183484:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767782.183487:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767782.183494:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00080000:0.0:1775767782.183500:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767782.183501:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349640700 x1862023076829184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183509:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183517:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767782.183518:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076829184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767782.183530:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.183533:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183539:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767782.183540:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076829568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767782.183561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767782.183566:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767782.183570:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767782.183576:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00100000:3.0:1775767782.183581:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349641c00 x1862023076829312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183583:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183592:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767782.183596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076829312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767782.183600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767782.183606:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.183609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183612:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767782.183618:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076829696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829440:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767782.183634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767782.183640:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767782.183648:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4283 00000800:00000100:3.0:1775767782.183649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767782.183656:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183661:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183667:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767782.183667:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349642680 x1862023076829440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767782.183683:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076829824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076829440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183708:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767782.183710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767782.183717:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767782.183717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767782.183721:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767782.183721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829440:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767782.183732:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00100000:3.0:1775767782.183735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829568:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767782.183738:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183744:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767782.183750:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767782.183760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767782.183766:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076829952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349640e00 x1862023076829568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183777:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767782.183787:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767782.183792:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767782.183801:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4283 00000100:00100000:3.0:1775767782.183806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076829568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767782.183808:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767782.183817:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767782.183819:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767782.183824:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767782.183824:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076829568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767782.183844:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076830080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767782.183845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767782.183893:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767782.183902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349641880 x1862023076829696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767782.183910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076829696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767782.183916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.183918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767782.183924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767782.183934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767782.183939:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349641500 x1862023076829824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767782.183946:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076829824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767782.183951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.183953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183955:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767782.183958:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767782.183967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767782.183972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349642a00 x1862023076829952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767782.183978:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076829952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767782.183983:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.183985:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.183987:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076829952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767782.183997:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767782.184006:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767782.184011:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767782/real 1775767782] req@ffffa04349642300 x1862023076830080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767782.184018:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076830080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767782.184023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767782.184042:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767782.184045:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767782.375095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767782.375096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767783.207117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767783.207225:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767783.399148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767783.399149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767784.231157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767784.231182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767784.423142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767784.423143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767785.255132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767785.255180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767785.447128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767785.447128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767786.279116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767786.279141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767786.471141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767786.471155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767787.303125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767787.303145:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767787.303153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775767787.303155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767787.303157:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767787.303168:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:0.0:1775767787.303189:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767787.303196:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767787.303232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076830208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767787.303262:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767787.303267:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767787.303269:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767787.303275:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:0.0:1775767787.303281:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767787.303286:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767787.303291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076830208:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775767787.303297:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767787.303315:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076830336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767787.303327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767787.303331:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767787.303333:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767787.303338:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:0.0:1775767787.303343:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767787.303348:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767787.303352:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767787.303358:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767787.303374:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076830464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767787.303383:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349640380 x1862023076830208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767787.303419:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076830208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303424:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767787.303429:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767787.303434:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767787.303437:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:3.0:1775767787.303441:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767787.303444:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767787.303445:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076830208:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767787.303454:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767787.303461:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767787.303467:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767787.303476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076830592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767787.303486:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767787.303495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349643100 x1862023076830336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303505:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767787.303510:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076830336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303513:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767787.303518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767787.303522:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767787.303526:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:3.0:1775767787.303540:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767787.303543:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767787.303544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767787.303552:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767787.303553:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830464:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767787.303562:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775767787.303572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767787.303581:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076830720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767787.303582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349640a80 x1862023076830464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.303608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076830464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303613:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767787.303630:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767787.303636:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767787.303636:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767787.303646:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767787.303648:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00100000:3.0:1775767787.303651:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767787.303657:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767787.303664:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767787.303666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830592:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775767787.303674:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767787.303706:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767787.303713:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076830848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767787.303715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349641180 x1862023076830592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.303747:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076830592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767787.303760:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767787.303769:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767787.303775:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767787.303775:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767787.303783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.303786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830592:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767787.303790:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00100000:3.0:1775767787.303790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830720:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767787.303798:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767787.303820:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767787.303827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767787.303839:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767787.303842:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349643b80 x1862023076830720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767787.303857:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076830976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767787.303859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076830720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767787.303888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767787.303893:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767787.303893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.303899:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076830720:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767787.303902:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767787.303905:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767787.303910:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:0.0:1775767787.303915:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767787.303919:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767787.303923:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767787.303931:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767787.303937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767787.303946:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076831104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767787.303949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349643800 x1862023076830848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.303979:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076830848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767787.303998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767787.304003:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767787.304003:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.304007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767787.304011:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767787.304014:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767787.304014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767787.304024:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4288 00000100:00080000:0.0:1775767787.304061:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767787.304067:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767787.304069:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767787.304077:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767787.304081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349640a80 x1862023076830976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.304094:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076830976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767787.304096:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076831232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767787.304141:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767787.304145:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.304149:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076830976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767787.304159:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076831104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767787.304171:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767787.304183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349643100 x1862023076831104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.304190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076831104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767787.304196:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767787.304197:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.304199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076831104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767787.304203:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076831232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767787.304212:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767787.304217:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767787/real 1775767787] req@ffffa04349640380 x1862023076831232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767787.304225:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076831232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767787.304230:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767787.304231:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767787.304234:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076831232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767787.495109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767787.495112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767788.327145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767788.327218:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767788.519150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767788.519150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767789.351136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767789.351156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767789.543095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767789.543097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767790.375108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767790.375170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767790.567127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767790.567127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767791.399110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767791.399147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767791.591090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767791.591090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767792.423118:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767792.423121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767792.423128:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767792.423132:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423140:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423149:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423155:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767792.423157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767792.423188:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076831360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423210:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423214:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767792.423215:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423218:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423221:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423223:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423231:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767792.423239:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076831488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423247:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423252:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767792.423252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076831360:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767792.423253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423261:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423263:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423265:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423268:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767792.423278:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076831616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423284:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423287:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767792.423288:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423291:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423293:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423295:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423299:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767792.423308:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076831744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767792.423314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767792.423320:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423323:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767792.423324:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423329:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423331:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423334:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423337:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767792.423344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076831872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767792.423352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423356:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423358:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423362:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767792.423366:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349642300 x1862023076831360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767792.423375:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076832000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423383:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423387:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767792.423387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076831360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767792.423390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423399:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423405:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767792.423407:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767792.423408:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767792.423417:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423422:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076831360:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767792.423424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076832128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423431:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423433:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767792.423434:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423437:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423439:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423441:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767792.423451:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767792.423453:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076832256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767792.423464:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767792.423466:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767792.423467:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767792.423470:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4294 00000100:00080000:0.0:1775767792.423472:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767792.423474:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767792.423477:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767792.423487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076832384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767792.423488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349642a00 x1862023076831488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.423522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076831488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.423529:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.423532:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423534:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767792.423539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767792.423557:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349641500 x1862023076831616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.423575:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076831616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.423581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.423586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423589:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767792.423593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767792.423611:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423620:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349641880 x1862023076831744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.423630:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076831744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.423639:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.423643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767792.423655:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767792.423693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349640e00 x1862023076831872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.423719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076831872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.423725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.423728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423731:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076831872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767792.423735:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767792.423764:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423776:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349642680 x1862023076832000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.423785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076832000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.423795:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.423799:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.423802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767792.423808:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767792.423827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.423998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349641c00 x1862023076832128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.424011:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076832128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.424021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.424025:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.424080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767792.424117:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767792.424140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.424152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349640700 x1862023076832256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.424165:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076832256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.424174:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.424178:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.424181:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767792.424187:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767792.424204:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767792.424212:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767792/real 1775767792] req@ffffa04349642300 x1862023076832384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767792.424220:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076832384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767792.424225:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767792.424227:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767792.424230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076832384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767792.615103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767792.615104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767793.447166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767793.447170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767793.639181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767793.639260:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767794.471122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767794.471204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767794.663134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767794.663140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767795.495117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767795.495123:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767795.687121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767795.687155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767796.519132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767796.519192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767796.711112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767796.711112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775767797.543100:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543108:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767797.543113:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543121:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543131:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767797.543133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767797.543137:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767797.543174:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076832512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767797.543204:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543208:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767797.543210:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543215:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543220:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543224:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543234:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767797.543234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076832512:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767797.543249:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076832640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767797.543263:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543265:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767797.543266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543269:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543272:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543274:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543277:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767797.543285:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076832768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767797.543291:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543294:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767797.543295:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543301:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4299 00000800:00000100:3.0:1775767797.543301:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767797.543306:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543309:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543312:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767797.543319:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076832896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767797.543325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543328:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767797.543329:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543331:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543335:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543337:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543340:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767797.543347:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349640380 x1862023076832512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767797.543350:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076833024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767797.543381:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767797.543384:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076832512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767797.543385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767797.543388:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543394:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543399:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543405:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767797.543409:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543417:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767797.543418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076833152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767797.543427:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076832512:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767797.543429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543436:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767797.543438:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543441:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543443:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543445:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543448:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767797.543455:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076833280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767797.543458:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832640:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767797.543475:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543478:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767797.543480:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767797.543485:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4299 00000800:00000100:3.0:1775767797.543488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767797.543492:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543499:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543503:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349643100 x1862023076832640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767797.543504:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767797.543518:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076832640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767797.543524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076833408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767797.543528:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543535:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832640:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767797.543546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767797.543552:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767797.543553:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767797.543555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832768:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767797.543559:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4299 00000100:00080000:0.0:1775767797.543564:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767797.543567:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767797.543571:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767797.543575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767797.543583:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076833536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767797.543587:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349640a80 x1862023076832768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543612:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076832768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543622:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767797.543628:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767797.543639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543645:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349643800 x1862023076832896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076832896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543668:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076832896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767797.543680:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767797.543694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543718:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349643b80 x1862023076833024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767797.543720:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767797.543736:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076833024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543744:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543746:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543749:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767797.543753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767797.543767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349641180 x1862023076833152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543781:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076833152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543788:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767797.543804:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767797.543814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349640380 x1862023076833280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076833280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543836:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543838:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767797.543843:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767797.543852:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543857:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349642300 x1862023076833408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076833408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543872:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767797.543875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767797.543884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767797.543888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767797/real 1775767797] req@ffffa04349640700 x1862023076833536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767797.543895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076833536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767797.543899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767797.543901:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767797.543903:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076833536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767797.735128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767797.735226:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767798.567127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767798.567392:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767798.759153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767798.759176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767799.591173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767799.591211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767799.783158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767799.783159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767800.615113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767800.615252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767800.807158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767800.807158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767801.639126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767801.639160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767801.831181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767801.831181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767802.663122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767802.663144:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767802.663158:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663171:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4304 00010000:00010000:3.0:1775767802.663174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767802.663184:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663194:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767802.663245:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076833664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663276:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663279:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767802.663281:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663284:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663288:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663293:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663304:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767802.663313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076833664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767802.663318:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076833792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663329:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767802.663330:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663333:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663335:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663338:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663342:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767802.663352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076833920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663360:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663363:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767802.663364:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663366:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663368:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663371:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663374:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767802.663383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076834048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663388:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767802.663392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767802.663393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767802.663396:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663400:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663403:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663406:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663409:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767802.663418:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076834176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663425:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767802.663427:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663430:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663434:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663437:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663441:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767802.663441:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349641c00 x1862023076833664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767802.663451:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076834304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663458:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767802.663459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076833664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767802.663463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767802.663467:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663476:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663480:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767802.663480:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767802.663486:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767802.663488:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775767802.663491:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767802.663495:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076833664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767802.663503:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076834432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767802.663509:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663512:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767802.663513:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767802.663515:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663518:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663528:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663530:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833792:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767802.663533:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767802.663547:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076834560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767802.663555:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767802.663557:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767802.663561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767802.663563:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767802.663568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349642680 x1862023076833792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767802.663569:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4304 00000100:00080000:0.0:1775767802.663574:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767802.663580:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767802.663587:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767802.663592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076833792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767802.663599:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076834688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767802.663599:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663606:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767802.663624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767802.663638:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663644:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349640e00 x1862023076833920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076833920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663691:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076833920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767802.663702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767802.663724:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663735:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349641880 x1862023076834048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076834048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767802.663770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767802.663785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663792:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349641500 x1862023076834176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076834176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767802.663814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767802.663824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663830:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349642a00 x1862023076834304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076834304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663844:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663846:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767802.663849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767802.663858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663863:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349640700 x1862023076834432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663869:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076834432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663882:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767802.663900:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767802.663909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349642300 x1862023076834560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076834560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663929:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663931:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767802.663934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767802.663944:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767802.663949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767802/real 1775767802] req@ffffa04349640380 x1862023076834688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767802.663958:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076834688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767802.663967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767802.663970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767802.663974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076834688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767802.855155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767802.855155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767803.687146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767803.687244:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767803.879159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767803.879159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767804.711151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767804.711183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767804.903162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767804.903175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767805.735170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767805.735219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767805.927155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767805.927196:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767806.759136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767806.759166:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767806.951126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767806.951160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767807.783130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767807.783172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767807.783179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767807.783188:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783195:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783204:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783208:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767807.783243:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076834816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767807.783273:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783276:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783279:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783295:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783303:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767807.783314:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076834816:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767807.783316:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076834944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767807.783331:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783335:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783338:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783340:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783343:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767807.783352:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076835072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783358:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767807.783362:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783364:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783367:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783369:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783372:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767807.783373:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767807.783383:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076835200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783389:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783398:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767807.783400:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783403:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783407:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783410:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783414:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350932680 x1862023076834816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767807.783416:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767807.783430:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076835328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767807.783437:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076834816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767807.783439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767807.783449:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783458:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783461:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783464:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767807.783464:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767807.783472:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767807.783475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767807.783480:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076835456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767807.783484:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076834816:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767807.783496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767807.783500:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767807.783516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834944:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767807.783527:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783530:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783533:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783541:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767807.783547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767807.783553:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076835584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767807.783564:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767807.783565:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783567:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:1.0:1775767807.783571:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767807.783574:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783579:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767807.783579:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350931c00 x1862023076834944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767807.783589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076835712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767807.783620:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767807.783622:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076834944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767807.783628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767807.783632:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767807.783643:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4309 00000100:00080000:3.0:1775767807.783643:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767807.783646:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767807.783650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767807.783653:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767807.783659:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767807.783659:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076834944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767807.783674:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076835840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767807.783675:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767807.783712:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350930a80 x1862023076835072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.783736:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076835072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.783745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.783749:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.783752:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767807.783758:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767807.783778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350930000 x1862023076835200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.783802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076835200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.783811:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.783814:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.783818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767807.783824:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767807.783843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783852:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350931880 x1862023076835328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.783864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076835328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.783873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.783876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.783879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076835328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767807.783885:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767807.783903:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350930e00 x1862023076835456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.783924:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076835456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.783932:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.783936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.783939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767807.783958:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767807.783980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.783990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350932300 x1862023076835584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.784002:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076835584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.784017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.784020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.784023:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767807.784050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767807.784070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.784081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350932a00 x1862023076835712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.784094:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076835712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.784101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.784104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.784107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767807.784113:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767807.784132:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767807.784142:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767807/real 1775767807] req@ffffa04350930700 x1862023076835840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767807.784154:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076835840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767807.784163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767807.784166:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767807.784169:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076835840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767807.975134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767807.975145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767808.807131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767808.807169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767808.999125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767808.999127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767809.831124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767809.831170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767810.023123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767810.023127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767810.855132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767810.855197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767811.047170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767811.047180:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767811.879129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767811.879226:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767812.071154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767812.071159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767812.903150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767812.903151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767812.903172:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903188:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903202:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903212:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775767812.903216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767812.903268:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076835968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767812.903296:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767812.903303:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903307:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903316:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767812.903328:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767812.903337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076835968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767812.903342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076836096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767812.903352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903355:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767812.903357:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903361:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903364:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903388:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767812.903393:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767812.903403:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076836224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767812.903411:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767812.903416:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767812.903417:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767812.903420:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903424:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903428:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767812.903432:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767812.903441:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076836352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767812.903449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903453:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767812.903454:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903458:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903462:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903465:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903466:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350930700 x1862023076835968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767812.903470:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767812.903479:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076836480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767812.903487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767812.903488:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076835968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767812.903496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767812.903498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767812.903499:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767812.903502:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:3.0:1775767812.903503:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767812.903506:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767812.903508:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076835968:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767812.903510:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767812.903515:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767812.903525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076836608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767812.903532:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767812.903534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767812.903539:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903543:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903546:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903550:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767812.903578:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767812.903582:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767812.903589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076836736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767812.903596:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350932a00 x1862023076836096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767812.903606:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767812.903609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076836096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767812.903610:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767812.903612:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903616:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00080000:1.0:1775767812.903620:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767812.903620:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767812.903624:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836096:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767812.903634:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767812.903636:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767812.903645:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076836864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767812.903654:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767812.903655:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767812.903658:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767812.903660:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767812.903664:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4314 00000100:00100000:3.0:1775767812.903664:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350932300 x1862023076836224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767812.903668:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767812.903672:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076836224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767812.903677:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767812.903685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767812.903686:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076836992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767812.903688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767812.903698:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767812.903717:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350930e00 x1862023076836352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076836352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903746:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903749:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767812.903755:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767812.903769:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903776:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350931880 x1862023076836480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076836480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903794:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076836480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767812.903805:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767812.903815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350930000 x1862023076836608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076836608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767812.903842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767812.903851:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350930a80 x1862023076836736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076836736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767812.903879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767812.903887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350931c00 x1862023076836864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076836864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903908:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767812.903911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767812.903920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767812.903925:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767812/real 1775767812] req@ffffa04350932680 x1862023076836992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767812.903931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076836992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767812.903936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767812.903938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767812.903940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076836992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767813.095178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767813.095178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767813.927161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767813.927245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767814.119166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767814.119166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767814.951095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767814.951145:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767815.143187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767815.143195:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767815.975125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767815.975168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767816.167115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767816.167161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767816.999153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767816.999172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767817.191171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767817.191171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767818.023116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767818.023131:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767818.023142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767818.023142:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023154:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775767818.023161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767818.023162:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767818.023200:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076837120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767818.023226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767818.023232:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023236:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023239:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023243:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767818.023274:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076837120:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767818.023284:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076837248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767818.023298:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767818.023303:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023308:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023312:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023316:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023321:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767818.023333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076837376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767818.023344:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767818.023345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767818.023351:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023356:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023360:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023365:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767818.023381:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076837504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767818.023390:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7d500 x1862023076837120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767818.023395:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767818.023407:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023418:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00100000:3.0:1775767818.023422:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076837120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767818.023425:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023433:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023442:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767818.023443:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.023452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767818.023459:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076837632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767818.023460:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076837120:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767818.023475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767818.023481:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023486:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00100000:3.0:1775767818.023490:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767818.023491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023499:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767818.023533:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076837760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767818.023538:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767818.023549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767818.023555:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7fb80 x1862023076837248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767818.023556:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023565:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023572:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023578:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023583:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767818.023583:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076837248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.023597:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767818.023598:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076837888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767818.023604:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.023613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767818.023623:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767818.023626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767818.023628:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767818.023632:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767818.023638:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023643:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023647:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767818.023653:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767818.023656:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767818.023670:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076838016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767818.023674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7ce00 x1862023076837376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767818.023692:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767818.023697:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767818.023699:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767818.023702:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076837376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767818.023706:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4319 00000100:00080000:2.0:1775767818.023712:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767818.023719:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767818.023720:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767818.023725:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767818.023727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.023733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767818.023739:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076838144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767818.023743:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767818.023775:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.023787:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7ed80 x1862023076837504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.023800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076837504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.023808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.023811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.023815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767818.023821:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767818.023839:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.023851:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7f800 x1862023076837632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.023870:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076837632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.023879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.023883:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.023887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076837632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767818.023905:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076837760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767818.023924:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.023935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7ea00 x1862023076837760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.023947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076837760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.023966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.023970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.023973:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076837760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767818.023979:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076837888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767818.023997:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.024008:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7df80 x1862023076837888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.024021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076837888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.024056:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.024059:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.024063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076837888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767818.024069:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767818.024086:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.024097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7d880 x1862023076838016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.024110:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076838016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.024119:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.024122:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.024125:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767818.024131:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767818.024149:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767818.024160:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767818/real 1775767818] req@ffffa0434aa7d500 x1862023076838144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767818.024171:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076838144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767818.024179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767818.024182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767818.024185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767818.215080:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767818.215115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767819.047098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767819.047146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767819.239166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767819.239168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767820.071109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767820.071173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767820.263123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767820.263123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767821.095124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767821.095159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767821.287131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767821.287132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767822.119113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767822.119193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767822.311106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767822.311108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767823.143137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767823.143155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775767823.143160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767823.143161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767823.143167:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767823.143175:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143198:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143204:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767823.143230:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076838272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143251:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767823.143254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767823.143256:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767823.143258:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143261:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143265:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143273:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767823.143280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076838272:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767823.143286:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076838400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767823.143297:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767823.143298:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767823.143300:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143302:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143304:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143307:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767823.143316:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076838528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767823.143329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767823.143331:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767823.143338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767823.143342:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143349:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143353:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143358:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767823.143367:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076838656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143375:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767823.143376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7e680 x1862023076838272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767823.143380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767823.143383:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767823.143392:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143396:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143398:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143401:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767823.143403:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076838272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767823.143411:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076838784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767823.143422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767823.143428:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767823.143430:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767823.143431:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767823.143436:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076838272:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767823.143437:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143443:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143446:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143471:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767823.143476:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767823.143484:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076838912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767823.143493:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767823.143496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767823.143499:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767823.143500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767823.143506:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143510:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767823.143515:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7c700 x1862023076838400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767823.143523:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767823.143534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076838400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767823.143538:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076839040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767823.143556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767823.143558:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767823.143563:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767823.143564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767823.143567:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767823.143570:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838400:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767823.143574:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143578:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767823.143579:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767823.143583:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767823.143588:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767823.143598:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076839168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767823.143603:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767823.143611:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767823.143614:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7dc00 x1862023076838528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767823.143615:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767823.143623:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767823.143626:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076838528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767823.143629:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4324 00000100:00080000:2.0:1775767823.143636:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767823.143638:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767823.143641:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767823.143645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767823.143648:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767823.143649:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767823.143658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767823.143662:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076839296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767823.143673:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143681:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7c380 x1862023076838656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076838656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767823.143702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767823.143712:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7c000 x1862023076838784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143724:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076838784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143733:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076838784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767823.143743:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767823.143753:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143767:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7f100 x1862023076838912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076838912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076838912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767823.143787:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767823.143798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143803:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7f480 x1862023076839040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076839040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767823.143826:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767823.143835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7e300 x1862023076839168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143848:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076839168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143854:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767823.143859:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767823.143867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767823.143872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767823/real 1775767823] req@ffffa0434aa7e680 x1862023076839296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767823.143883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076839296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767823.143893:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767823.143895:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767823.143897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076839296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767823.335125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767823.335126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767824.167126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767824.167161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767824.359104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767824.359105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767825.191130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767825.191156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767825.383115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767825.383115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767826.215125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767826.215173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767826.407097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767826.407105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767827.239142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767827.239147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767827.431135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767827.431135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767828.263132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767828.263144:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263153:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263163:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263168:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767828.263203:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076839424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775767828.263205:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767828.263224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767828.263230:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263233:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263235:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263238:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263245:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767828.263255:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076839552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767828.263261:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076839424:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767828.263276:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263278:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767828.263280:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263283:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263286:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263288:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263293:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767828.263302:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076839680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767828.263315:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767828.263319:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263321:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263324:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767828.263329:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767828.263332:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767828.263352:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076839808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767828.263363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263366:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767828.263368:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767828.263374:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7d500 x1862023076839424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767828.263375:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263391:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263396:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263401:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767828.263401:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076839424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767828.263431:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076839936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767828.263435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767828.263443:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767828.263443:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767828.263447:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767828.263448:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076839424:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767828.263449:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263458:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263461:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263464:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263468:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767828.263474:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767828.263478:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076840064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767828.263487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263489:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767828.263491:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767828.263500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767828.263503:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263507:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263510:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263513:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767828.263514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7d880 x1862023076839552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767828.263526:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076840192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767828.263540:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076839552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.263569:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767828.263570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767828.263576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767828.263579:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767828.263579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767828.263587:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00100000:3.0:1775767828.263587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839552:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767828.263594:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263599:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767828.263599:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839680:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767828.263606:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767828.263618:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076840320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767828.263625:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767828.263630:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767828.263636:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767828.263637:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767828.263640:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4329 00000100:00080000:2.0:1775767828.263643:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767828.263646:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767828.263650:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767828.263662:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076840448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767828.263662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7df80 x1862023076839680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.263687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076839680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.263695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.263698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.263701:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767828.263707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767828.263730:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.263742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7ea00 x1862023076839808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.263757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076839808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.263770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.263774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.263778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767828.263785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767828.263806:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.263823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7f800 x1862023076839936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.263837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076839936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.263846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.263850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.263853:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076839936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767828.263876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767828.263905:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.263921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7ed80 x1862023076840064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.263939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076840064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.263949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.263953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.263959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767828.263976:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767828.264068:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000800:00000100:3.0:1775767828.264071:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.264090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7ce00 x1862023076840192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.264213:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076840192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.264223:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.264228:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.264232:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767828.264281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767828.264308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.264321:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7fb80 x1862023076840320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.264333:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076840320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.264339:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.264342:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.264345:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767828.264352:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767828.264372:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767828.264384:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767828/real 1775767828] req@ffffa0434aa7d500 x1862023076840448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767828.264397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076840448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767828.264417:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767828.264419:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767828.264422:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076840448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767828.455095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767828.455192:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767829.287163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767829.287197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767829.479125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767829.479125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767830.311135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767830.311211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767830.503088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767830.503116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767831.335130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767831.335150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767831.527140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767831.527179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767832.359102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767832.359155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767832.551119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767832.551119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775767833.383118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383125:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767833.383129:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383140:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383149:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775767833.383149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767833.383154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767833.383155:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767833.383191:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076840576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767833.383216:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383219:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383221:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383224:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383232:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767833.383242:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076840704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767833.383251:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076840576:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767833.383253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767833.383263:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383269:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383273:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383275:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383279:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767833.383303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076840832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767833.383313:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383316:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383320:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383323:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383326:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767833.383336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076840960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767833.383347:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767833.383349:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383367:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383371:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383374:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383380:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767833.383394:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076841088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767833.383401:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767833.383405:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767833.383411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383421:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383427:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383430:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383435:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767833.383439:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767833.383447:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076841216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383456:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767833.383461:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841088:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775767833.383464:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383469:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383473:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383475:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383479:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767833.383489:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076841344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767833.383493:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841216:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767833.383501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767833.383503:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767833.383504:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383508:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383510:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383513:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383516:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767833.383523:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076841472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767833.383534:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767833.383535:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767833.383537:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767833.383541:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767833.383549:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4335 00000100:00080000:2.0:1775767833.383552:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767833.383555:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767833.383559:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767833.383568:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076841600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767833.383581:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767833.383604:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767833.575090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767833.575091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775767834.407179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767834.407199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767834.599136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767834.599136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767835.431121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767835.431148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767835.623108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767835.623108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767836.455104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767836.455272:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775767836.455349:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775767836.455394:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775767836.455411:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455460:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455474:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455484:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455493:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455503:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455514:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775767836.455526:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767836.455534:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7d500 x1862023076841600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1775767836.455537:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767836.455565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076841600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455584:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455590:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767836.455600:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7dc00 x1862023076841344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076841344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455703:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455710:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767836.455717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7c380 x1862023076841216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076841216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455744:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455747:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767836.455758:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7f100 x1862023076840960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076840960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455777:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767836.455791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7c000 x1862023076841088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076841088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455814:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767836.455824:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7e300 x1862023076840704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076840704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767836.455864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7f480 x1862023076840832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076840832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455890:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076840832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767836.455902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7e680 x1862023076840576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076840576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076840576:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767836.455959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767833/real 1775767836] req@ffffa0434aa7c700 x1862023076841472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767836.455970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076841472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767836.455978:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767836.455981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767836.455984:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076841472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767836.647146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767836.647147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767837.479111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767837.479156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767837.671108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767837.671126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767838.503137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767838.503137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767838.503153:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503163:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503172:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503177:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767838.503217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076841728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767838.503240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767838.503244:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503247:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503250:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503252:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767838.503270:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076841856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767838.503280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076841728:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767838.503280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767838.503291:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503295:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503298:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503301:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503315:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767838.503329:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076841984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767838.503339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767838.503345:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503348:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503355:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503360:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767838.503371:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767838.503394:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076842112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767838.503406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767838.503411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503417:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00100000:2.0:1775767838.503422:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825ea00 x1862023076841728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767838.503424:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503433:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503443:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767838.503454:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076841728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767838.503459:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076842240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767838.503474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503479:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767838.503480:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767838.503482:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767838.503488:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767838.503490:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503495:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503500:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503506:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767838.503520:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076842368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767838.503528:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076841728:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767838.503535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767838.503540:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767838.503542:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503549:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503554:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503559:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503565:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767838.503568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767838.503581:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076842496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767838.503601:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767838.503602:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767838.503609:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767838.503611:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503617:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503623:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767838.503625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825ed80 x1862023076841856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767838.503637:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503644:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767838.503645:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076841856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.503659:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767838.503664:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076842624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767838.503667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.503676:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767838.503682:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767838.503687:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767838.503688:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767838.503693:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767838.503700:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4340 00000100:00080000:3.0:1775767838.503704:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767838.503710:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767838.503715:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767838.503721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767838.503732:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076842752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767838.503739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825c000 x1862023076841984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.503754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076841984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.503762:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.503767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.503771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076841984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767838.503779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076842112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767838.503808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.503832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825e680 x1862023076842112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.503847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076842112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.503858:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:1.0:1775767838.503861:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767838.503863:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.503869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076842112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767838.503878:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076842240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767838.503899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.503911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825f100 x1862023076842240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.503923:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076842240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.503933:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.503936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.503939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076842240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767838.503946:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767838.503964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.503973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825e300 x1862023076842368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.503984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076842368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.503996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.504000:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.504002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767838.504021:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767838.504070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.504082:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825d500 x1862023076842496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.504093:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076842496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.504102:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.504105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.504109:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767838.504113:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767838.504133:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.504143:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825dc00 x1862023076842624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.504154:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076842624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.504162:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.504164:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.504175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767838.504180:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767838.504201:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767838.504211:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767838/real 1775767838] req@ffffa0435825d880 x1862023076842752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767838.504222:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076842752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767838.504230:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767838.504233:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767838.504236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076842752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767838.695091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767838.695091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767839.527103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767839.527167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767839.719110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767839.719110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767840.551170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767840.551312:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767840.743192:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767840.743194:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767841.575152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767841.575198:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767841.767111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767841.767127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767842.599125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767842.599164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767842.791143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767842.791143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:3.0:1775767843.623151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767843.623153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767843.623165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767843.623170:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623184:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623198:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623223:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767843.623293:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076842880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767843.623329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767843.623340:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623348:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623355:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623362:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623377:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767843.623391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076842880:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767843.623400:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076843008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767843.623418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767843.623426:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623432:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623438:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623444:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623452:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767843.623468:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076843136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767843.623474:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767843.623485:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623490:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767843.623492:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623498:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623503:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623509:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767843.623527:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825ce00 x1862023076842880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767843.623532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076843264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767843.623549:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1775767843.623558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076842880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767843.623558:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623565:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623571:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623576:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767843.623577:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775767843.623583:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767843.623599:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076843392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767843.623603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.623611:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076842880:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767843.623614:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767843.623622:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623629:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623634:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623640:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767843.623645:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843008:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767843.623648:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767843.623663:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076843520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767843.623679:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767843.623684:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767843.623685:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767843.623689:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623695:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00080000:3.0:1775767843.623701:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767843.623704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825d880 x1862023076843008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767843.623707:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623715:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767843.623728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076843008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767843.623731:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076843648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767843.623745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767843.623748:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767843.623751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767843.623754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767843.623757:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767843.623758:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843008:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767843.623764:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00100000:2.0:1775767843.623768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843136:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767843.623769:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623775:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623782:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767843.623797:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076843776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767843.623798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.623815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825dc00 x1862023076843136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767843.623815:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767843.623821:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767843.623824:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767843.623831:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4345 00000100:00100000:2.0:1775767843.623837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076843136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767843.623837:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767843.623842:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767843.623850:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767843.623852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.623858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.623865:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767843.623865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076843904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767843.623876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767843.623924:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.623945:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825d500 x1862023076843264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.623969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076843264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.623984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.623991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00010000:00010000:0.0:1775767843.623996:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767843.623999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767843.624015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767843.624081:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.624097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825e300 x1862023076843392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.624111:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076843392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.624121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.624125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.624128:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076843392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767843.624149:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767843.624170:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.624180:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825f100 x1862023076843520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.624213:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076843520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.624223:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.624226:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.624230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767843.624236:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767843.624285:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.624304:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825e680 x1862023076843648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.624318:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076843648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.624327:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.624330:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.624334:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767843.624340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767843.624360:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.624369:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825c000 x1862023076843776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.624382:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076843776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.624406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.624410:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.624413:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767843.624420:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767843.624442:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767843.624452:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767843/real 1775767843] req@ffffa0435825ed80 x1862023076843904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767843.624465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076843904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767843.624474:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767843.624477:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767843.624480:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076843904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767843.815110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767843.815116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767844.647111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767844.647191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767844.839107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767844.839107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767845.671186:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767845.671269:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767845.863091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767845.863203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767846.695200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767846.695343:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767846.887176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767846.887187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767847.719287:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767847.719352:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767847.911105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767847.911105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767848.743191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767848.743223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767848.743235:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743249:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743261:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743269:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767848.743320:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076844032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767848.743350:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743354:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743357:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743361:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743371:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767848.743384:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076844160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767848.743386:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076844032:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767848.743399:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743402:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767848.743403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743406:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743408:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743410:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743413:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767848.743421:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076844288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743427:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743429:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767848.743430:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743433:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743465:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743469:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743475:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767848.743485:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767848.743487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076844416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767848.743501:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743504:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743507:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743510:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743514:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767848.743523:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076844544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767848.743537:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825ea00 x1862023076844032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767848.743540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743547:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767848.743551:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743560:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743563:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743567:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767848.743586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076844032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767848.743589:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767848.743598:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076844672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743604:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743611:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767848.743614:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743622:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743627:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767848.743629:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767848.743630:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743635:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767848.743636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.743642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076844032:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767848.743644:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076844800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743652:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743654:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767848.743655:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743662:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743665:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767848.743669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767848.743670:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743677:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767848.743688:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076844928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767848.743696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767848.743701:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1775767848.743703:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767848.743703:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767848.743710:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4350 00000100:00080000:3.0:1775767848.743714:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767848.743717:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767848.743721:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767848.743726:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825ed80 x1862023076844160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767848.743730:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076845056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767848.743762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076844160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.743773:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.743777:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.743782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767848.743791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767848.743815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.743829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825c000 x1862023076844288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1775767848.743833:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767848.743844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076844288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.743869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.743873:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.743877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767848.743893:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767848.743920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.743935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825e680 x1862023076844416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.743947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076844416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.743954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.743957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.743959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767848.743964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767848.743984:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.743997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825f100 x1862023076844544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.744012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076844544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.744023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.744028:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.744032:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076844544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767848.744040:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767848.744066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.744125:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825e300 x1862023076844672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.744142:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076844672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.744149:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.744151:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.744154:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767848.744158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767848.744190:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.744197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825d500 x1862023076844800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.744205:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076844800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.744226:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.744228:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.744230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767848.744246:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767848.744284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.744297:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825dc00 x1862023076844928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.744311:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076844928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.744336:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.744340:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.744343:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076844928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767848.744349:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767848.744367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767848.744375:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767848/real 1775767848] req@ffffa0435825d880 x1862023076845056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767848.744383:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076845056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767848.744390:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767848.744392:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767848.744394:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767848.935185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767848.935185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767849.767145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767849.767502:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767849.959145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767849.959151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767850.791139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767850.791269:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767850.983123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767850.983174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767851.815132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767851.815172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767852.007154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767852.007200:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767852.839145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767852.839210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767853.031130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767853.031187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767853.863106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767853.863146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767853.863151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767853.863181:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863192:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863202:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863209:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767853.863244:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076845184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767853.863270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767853.863276:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863281:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863285:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863290:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863300:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767853.863301:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076845184:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767853.863324:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076845312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767853.863357:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863361:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767853.863364:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767853.863364:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767853.863373:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863380:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863384:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863391:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767853.863408:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076845440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767853.863410:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350932680 x1862023076845184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767853.863435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863439:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767853.863441:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863446:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00100000:2.0:1775767853.863447:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076845184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767853.863450:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863455:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863462:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767853.863469:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767853.863477:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076845568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767853.863482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076845184:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767853.863494:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863496:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767853.863498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863501:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00100000:2.0:1775767853.863509:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767853.863511:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863517:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863521:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767853.863532:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076845696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767853.863537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767853.863547:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863550:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767853.863552:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767853.863552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350931c00 x1862023076845312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767853.863560:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863565:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767853.863570:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863574:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767853.863574:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076845312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767853.863586:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076845824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767853.863586:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863596:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767853.863599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767853.863603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767853.863604:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767853.863606:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863612:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863614:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767853.863618:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845440:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767853.863619:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863625:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767853.863634:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076845952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767853.863650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767853.863659:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767853.863662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350930a80 x1862023076845440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767853.863663:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767853.863666:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863673:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863679:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767853.863682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076845440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767853.863686:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863696:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767853.863696:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.863705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767853.863716:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076846080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767853.863722:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845568:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767853.863736:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767853.863740:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767853.863743:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767853.863749:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4355 00000100:00080000:1.0:1775767853.863754:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767853.863754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767853.863762:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767853.863769:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767853.863772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350930000 x1862023076845568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767853.863782:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076846208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767853.863790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076845568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.863798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.863802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767853.863817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767853.863832:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767853.863848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350931880 x1862023076845696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767853.863856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076845696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.863862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.863866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076845696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767853.863870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767853.863888:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767853.863898:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350930e00 x1862023076845824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767853.863918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076845824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.863927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863930:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.863933:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767853.863938:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767853.863954:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767853.863961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350932300 x1862023076845952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767853.863969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076845952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.863974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.863976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.863978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076845952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767853.863981:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767853.863991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767853.863996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350932a00 x1862023076846080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767853.864003:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076846080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.864017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.864019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.864021:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767853.864043:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767853.864059:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767853.864067:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767853/real 1775767853] req@ffffa04350930700 x1862023076846208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767853.864074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076846208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767853.864079:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767853.864081:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767853.864083:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767854.055143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767854.055143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767854.887124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767854.887179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767855.079170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767855.079179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775767855.911171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767855.911179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767856.103115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767856.103115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767856.935165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767856.935337:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767857.127122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767857.127122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767857.959145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767857.959194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767858.151127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767858.151140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767858.983121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767858.983133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767858.983142:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983157:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983166:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983174:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767858.983187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767858.983210:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076846336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767858.983232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767858.983236:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983239:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983241:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983244:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767858.983261:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076846464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767858.983268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076846336:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767858.983278:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983280:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767858.983282:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983285:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983287:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983290:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983293:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767858.983304:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076846592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767858.983310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767858.983313:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983316:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983325:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767858.983332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767858.983333:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983338:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767858.983348:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076846720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767858.983354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767858.983357:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983360:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983363:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983365:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983369:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767858.983375:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350930700 x1862023076846336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767858.983378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076846848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767858.983406:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983409:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767858.983410:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983413:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983415:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983417:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983423:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767858.983439:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076846976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767858.983445:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076846336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767858.983449:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767858.983459:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983464:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:2.0:1775767858.983464:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767858.983470:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767858.983474:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767858.983477:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076846336:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767858.983482:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767858.983493:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076847104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767858.983498:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767858.983501:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767858.983502:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767858.983504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767858.983508:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00080000:1.0:1775767858.983511:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983513:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767858.983517:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767858.983525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076847232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767858.983534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767858.983543:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767858.983548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350932a00 x1862023076846464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767858.983549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767858.983553:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767858.983565:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4360 00000100:00100000:2.0:1775767858.983569:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076846464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767858.983570:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767858.983579:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767858.983582:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775767858.983588:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767858.983591:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767858.983597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767858.983603:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076847360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767858.983614:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350932300 x1862023076846592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983632:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076846592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983648:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767858.983655:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767858.983668:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350930e00 x1862023076846720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983681:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076846720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767858.983693:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767858.983703:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983708:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350931880 x1862023076846848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983715:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076846848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983722:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076846848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767858.983734:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767858.983747:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350930000 x1862023076846976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983761:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076846976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983766:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983770:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076846976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767858.983773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767858.983783:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983787:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350930a80 x1862023076847104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076847104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767858.983823:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767858.983834:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350931c00 x1862023076847232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076847232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983859:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983861:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767858.983866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767858.983876:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767858.983882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767858/real 1775767858] req@ffffa04350932680 x1862023076847360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767858.983889:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076847360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767858.983893:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767858.983895:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767858.983897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076847360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767859.175161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767859.175162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767860.007185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767860.007273:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767860.199207:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767860.199209:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767861.031119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767861.031167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767861.223138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767861.223138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767862.055199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767862.055294:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767862.247199:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767862.247200:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767863.079179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767863.079301:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767863.271118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767863.271162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767864.103200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767864.103221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767864.103239:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103251:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4365 00010000:00010000:2.0:1775767864.103262:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767864.103265:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103276:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767864.103316:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076847488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767864.103360:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103364:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103378:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103387:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103396:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767864.103400:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076847488:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767864.103411:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076847616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103420:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103423:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767864.103424:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103439:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103442:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103445:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767864.103456:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076847744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103463:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103465:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767864.103466:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103468:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103471:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103474:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103478:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767864.103487:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076847872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103496:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767864.103501:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103505:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103510:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103514:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103521:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1775767864.103527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767864.103537:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076848000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103550:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103554:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767864.103557:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103562:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103565:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103568:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103573:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767864.103587:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076848128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767864.103588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350932680 x1862023076847488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767864.103651:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103653:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767864.103655:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103661:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103665:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767864.103669:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076847488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767864.103671:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103679:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767864.103691:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076848256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103699:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767864.103699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767864.103711:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767864.103713:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767864.103718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767864.103723:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103729:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767864.103735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076847488:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767864.103737:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767864.103745:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767864.103755:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076848384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767864.103763:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767864.103765:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767864.103766:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767864.103768:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4365 00000100:00080000:1.0:1775767864.103772:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767864.103781:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767864.103782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847616:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767864.103789:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767864.103803:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076848512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767864.103826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.103845:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350931c00 x1862023076847616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.103860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076847616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.103870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.103874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.103881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767864.103891:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767864.103925:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.103941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350930a80 x1862023076847744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.103989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076847744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.103999:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104007:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104012:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767864.104023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767864.104100:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104121:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350930000 x1862023076847872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076847872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104157:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104163:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104170:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076847872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767864.104178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767864.104206:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104230:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350931880 x1862023076848000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104255:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076848000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104278:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104283:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104287:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767864.104293:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767864.104319:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104332:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350930e00 x1862023076848128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104347:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076848128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104357:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104362:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104365:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767864.104373:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767864.104403:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104427:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350932300 x1862023076848256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104444:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076848256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104457:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104462:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104475:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767864.104486:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767864.104514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104527:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350932a00 x1862023076848384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104541:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076848384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104550:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104557:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767864.104582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767864.104605:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767864.104617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767864/real 1775767864] req@ffffa04350930700 x1862023076848512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767864.104631:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076848512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767864.104640:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767864.104644:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767864.104647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076848512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767864.295126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767864.295128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775767865.127146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767865.127231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767865.319143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767865.319143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767866.151101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767866.151186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767866.343146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767866.343147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767867.175153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767867.175214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767867.367098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767867.367099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767868.199230:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767868.199234:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767868.391154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767868.391185:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775767869.223152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767869.223177:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223183:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767869.223187:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223197:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223207:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223214:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767869.223262:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076848640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775767869.223266:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767869.223289:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223292:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767869.223294:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223297:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223300:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223302:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223312:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767869.223321:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076848768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767869.223335:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076848640:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767869.223336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767869.223346:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223350:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223352:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223354:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223375:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767869.223385:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076848896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767869.223392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223395:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767869.223397:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223399:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223409:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223412:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767869.223423:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076849024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767869.223432:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767869.223433:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223438:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767869.223439:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223445:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223447:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223450:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767869.223463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076849152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767869.223469:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223471:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767869.223478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223481:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223485:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223489:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767869.223495:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7fb80 x1862023076848640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767869.223505:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076849280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767869.223530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767869.223537:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223539:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223542:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223544:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223548:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767869.223552:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076848640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767869.223558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076849408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767869.223578:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223581:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767869.223581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767869.223582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223589:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223591:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767869.223593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767869.223595:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767869.223603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076848640:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767869.223621:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076849536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767869.223628:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767869.223631:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767869.223632:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767869.223635:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4370 00000100:00080000:2.0:1775767869.223637:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767869.223640:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767869.223643:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767869.223650:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076849664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767869.223655:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767869.223689:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.223705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7ce00 x1862023076848768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.223718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076848768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.223728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.223732:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.223735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767869.223742:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767869.223763:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.223775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7ed80 x1862023076848896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.223789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076848896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.223804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.223808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.223811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076848896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767869.223818:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767869.223839:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.223850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7f800 x1862023076849024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.223864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076849024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.223873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.223877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.223880:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767869.223886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767869.223905:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.223915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7ea00 x1862023076849152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.223927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076849152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.223935:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.223938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.223942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767869.223948:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767869.223966:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.223975:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7df80 x1862023076849280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.223987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076849280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.223996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.223999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.224003:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767869.224023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767869.224066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.224076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7d880 x1862023076849408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.224089:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076849408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.224097:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.224100:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.224103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767869.224109:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767869.224127:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.224138:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7c700 x1862023076849536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.224151:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076849536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.224159:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.224163:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.224166:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767869.224191:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767869.224211:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767869.224221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767869/real 1775767869] req@ffffa0434aa7e680 x1862023076849664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767869.224234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076849664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767869.224243:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767869.224246:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767869.224250:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076849664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767869.415101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767869.415127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767870.247200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767870.247236:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767870.439133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767870.439133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767871.271169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767871.271254:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767871.463138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767871.463139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767872.295170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767872.295207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767872.487113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767872.487113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767873.319176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767873.319190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767873.511127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767873.511127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767874.343174:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.343186:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767874.343190:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767874.343190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767874.343193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767874.343215:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.343225:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.343232:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767874.344305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076849792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767874.344350:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344353:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344357:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344361:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767874.344380:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076849920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344399:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344402:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767874.344404:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076849792:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767874.344405:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344412:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344416:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344419:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344422:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767874.344432:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076850048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344438:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767874.344442:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344444:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344457:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344460:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767874.344470:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767874.344482:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076850176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344491:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767874.344492:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344494:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344498:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344501:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344505:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767874.344508:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7f480 x1862023076849792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767874.344516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076850304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344525:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344528:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767874.344529:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076849792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767874.344530:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344538:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344544:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344548:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767874.344549:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767874.344553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767874.344556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344561:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076849792:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767874.344564:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076850432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344571:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344574:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767874.344575:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767874.344582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767874.344584:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344589:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344593:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344597:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767874.344606:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767874.344609:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076850560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767874.344617:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767874.344617:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7e300 x1862023076849920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767874.344625:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767874.344629:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344645:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00100000:3.0:1775767874.344648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076849920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767874.344649:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344655:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344661:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767874.344664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767874.344671:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076850688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767874.344671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344678:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076849920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767874.344689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767874.344690:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767874.344698:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767874.344699:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767874.344703:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4376 00000100:00080000:2.0:1775767874.344706:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767874.344709:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767874.344714:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767874.344714:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767874.344726:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076850816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767874.344727:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7c000 x1862023076850048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.344757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076850048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.344763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.344765:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767874.344772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767874.344784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.344790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7f100 x1862023076850176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.344797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076850176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.344802:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.344804:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344805:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767874.344814:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767874.344825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.344831:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7c380 x1862023076850304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.344838:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076850304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.344862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.344864:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076850304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767874.344870:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767874.344890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.344897:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7dc00 x1862023076850432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.344904:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076850432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.344909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.344911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767874.344924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767874.344934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.344939:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7d500 x1862023076850560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.344946:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076850560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.344951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.344952:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.344955:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767874.344971:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767874.344982:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.344998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7f480 x1862023076850688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.345005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076850688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.345010:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.345012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.345014:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767874.345017:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767874.345043:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767874.345050:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767874/real 1775767874] req@ffffa0434aa7e680 x1862023076850816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767874.345057:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076850816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767874.345062:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767874.345064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767874.345066:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076850816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767874.535096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767874.535113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767875.367126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767875.367151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767875.559128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767875.559128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767876.391126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767876.391161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767876.583141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767876.583141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767877.415106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767877.415180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767877.607128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767877.607129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767878.439105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767878.439152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767878.631111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767878.631117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767879.463127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767879.463163:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463179:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767879.463184:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775767879.463190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767879.463195:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463204:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463210:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767879.463248:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076850944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767879.463268:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463271:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767879.463272:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767879.463275:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463277:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767879.463287:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767879.463306:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076851072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767879.463309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076850944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767879.463316:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463318:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767879.463319:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767879.463323:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463325:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463330:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767879.463358:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767879.463368:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076851200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767879.463375:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767879.463379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767879.463388:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4381 00000800:00000100:3.0:1775767879.463392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767879.463393:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463398:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767879.463402:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767879.463411:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076851328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767879.463417:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767879.463421:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767879.463424:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463426:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463431:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463431:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7c700 x1862023076850944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767879.463435:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767879.463447:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076851456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767879.463453:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076850944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767879.463460:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463468:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767879.463471:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767879.463475:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767879.463477:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767879.463483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00100000:3.0:1775767879.463484:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076850944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767879.463489:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463494:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767879.463497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767879.463506:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767879.463513:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076851584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767879.463522:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463525:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767879.463528:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767879.463528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767879.463534:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463538:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463543:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463544:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7d880 x1862023076851072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767879.463549:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767879.463561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076851072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767879.463564:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076851712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767879.463587:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767879.463588:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463593:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767879.463595:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767879.463596:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767879.463601:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767879.463603:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463608:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463615:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851200:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767879.463617:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767879.463632:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076851840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767879.463639:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767879.463642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767879.463643:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767879.463647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767879.463648:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4381 00000100:00080000:2.0:1775767879.463654:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767879.463657:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767879.463661:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767879.463661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7df80 x1862023076851200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767879.463674:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076851968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767879.463676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076851200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767879.463710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767879.463723:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463729:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7ea00 x1862023076851328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463736:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076851328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463741:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463743:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767879.463748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767879.463758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7f800 x1862023076851456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076851456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463778:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076851456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767879.463790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767879.463800:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7ed80 x1862023076851584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463813:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076851584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463818:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767879.463825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767879.463834:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7ce00 x1862023076851712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076851712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463854:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463865:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463867:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767879.463871:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767879.463881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7fb80 x1862023076851840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076851840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463923:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767879.463930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767879.463951:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767879.463962:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767879/real 1775767879] req@ffffa0434aa7c700 x1862023076851968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767879.463983:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076851968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767879.463989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767879.463990:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767879.463993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076851968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767879.655147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767879.655147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767880.487142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767880.487174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767880.679154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767880.679165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767881.511090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767881.511150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767881.703130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767881.703133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767882.535127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767882.535149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767882.727102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767882.727102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775767883.559165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767883.559172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767883.751118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767883.751124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767884.583127:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767884.583136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767884.583140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767884.583143:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767884.583148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583157:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583165:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583182:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767884.583219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076852096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767884.583238:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583241:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767884.583242:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583245:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583262:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583270:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076852096:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775767884.583276:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767884.583288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076852224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767884.583297:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583299:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767884.583301:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583310:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583319:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583323:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767884.583332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076852352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767884.583333:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767884.583342:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583345:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767884.583346:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583349:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583351:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583354:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583358:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767884.583365:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076852480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767884.583381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7e680 x1862023076852096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767884.583384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583392:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767884.583395:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583400:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583403:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767884.583407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076852096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767884.583408:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583414:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775767884.583421:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767884.583426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076852608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767884.583430:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767884.583439:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767884.583439:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076852096:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767884.583444:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767884.583448:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583458:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583460:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583462:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583466:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767884.583470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767884.583476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076852736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767884.583484:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583487:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767884.583488:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583491:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583499:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767884.583500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767884.583506:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767884.583515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7f480 x1862023076852224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767884.583522:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076852864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767884.583532:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076852224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767884.583533:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767884.583543:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583551:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583556:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583562:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767884.583563:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767884.583565:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767884.583569:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767884.583575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076852992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767884.583575:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767884.583587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852352:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767884.583590:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767884.583593:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767884.583595:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767884.583598:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4386 00000100:00080000:2.0:1775767884.583602:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767884.583607:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767884.583608:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775767884.583614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767884.583620:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7d500 x1862023076852352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767884.583625:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076853120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767884.583638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076852352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583675:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583678:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767884.583683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767884.583696:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7dc00 x1862023076852480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076852480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583716:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583718:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767884.583722:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767884.583731:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7c380 x1862023076852608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076852608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583750:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583752:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076852608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767884.583763:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767884.583773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7f100 x1862023076852736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076852736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583794:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767884.583798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767884.583807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583812:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7c000 x1862023076852864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583819:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076852864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583823:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767884.583830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767884.583838:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583844:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7e300 x1862023076852992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076852992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583880:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583883:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076852992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767884.583889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076853120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767884.583910:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767884.583920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767884/real 1775767884] req@ffffa0434aa7e680 x1862023076853120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767884.583928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076853120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767884.583934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767884.583936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767884.583939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076853120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767884.775127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767884.775128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767885.607100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767885.607147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767885.799112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767885.799112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767886.631103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767886.631120:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767886.823100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767886.823100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767887.655103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767887.655162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767887.847111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767887.847113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767888.679112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767888.679152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767888.871083:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767888.871084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767889.703138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767889.703144:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703152:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767889.703156:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703166:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703176:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703184:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767889.703237:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076853248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703261:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703264:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767889.703266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703271:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703275:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703290:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767889.703291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076853248:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767889.703303:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076853376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767889.703317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703321:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703324:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703327:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703331:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767889.703339:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076853504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703347:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767889.703349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767889.703351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767889.703352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703355:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703358:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703361:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767889.703374:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076853632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703383:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703386:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767889.703388:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703392:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00100000:2.0:1775767889.703393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825ce00 x1862023076853248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767889.703395:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703398:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703403:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767889.703413:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076853248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767889.703413:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076853760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703422:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767889.703424:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767889.703426:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767889.703428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767889.703428:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703431:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00100000:2.0:1775767889.703432:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076853248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767889.703434:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703437:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703441:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767889.703449:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076853888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767889.703452:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767889.703459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767889.703463:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703467:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703470:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767889.703473:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767889.703473:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767889.703485:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825d880 x1862023076853376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767889.703487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076854016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703497:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767889.703498:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076853376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767889.703500:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767889.703502:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703505:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:2.0:1775767889.703508:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767889.703508:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767889.703511:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767889.703512:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703515:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853376:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767889.703516:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767889.703521:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767889.703524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076854144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767889.703532:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767889.703535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1775767889.703537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767889.703537:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767889.703540:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4391 00000100:00080000:3.0:1775767889.703543:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767889.703562:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767889.703566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767889.703575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076854272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767889.703578:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825dc00 x1862023076853504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076853504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767889.703612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767889.703632:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00010000:00010000:0.0:1775767889.703646:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767889.703646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825d500 x1862023076853632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076853632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703677:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703679:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767889.703684:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767889.703699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825e300 x1862023076853760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076853760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703753:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703757:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076853760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767889.703778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076853888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767889.703801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825f100 x1862023076853888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076853888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076853888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767889.703862:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767889.703882:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703890:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825e680 x1862023076854016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703897:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076854016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703912:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767889.703916:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767889.703929:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825c000 x1862023076854144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076854144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767889.703955:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767889.703963:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767889.703977:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767889/real 1775767889] req@ffffa0435825ed80 x1862023076854272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767889.703984:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076854272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767889.703989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767889.703991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767889.703993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076854272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767889.895108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767889.895109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767890.727079:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767890.727149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767890.919133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767890.919133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767891.751136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767891.751203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767891.943096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767891.943097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767892.775092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767892.775240:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767892.967129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767892.967129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767893.799123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767893.799192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767893.991112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767893.991120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767894.823163:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823178:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767894.823184:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767894.823192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767894.823198:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:1.0:1775767894.823213:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775767894.823219:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767894.823223:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767894.823277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076854400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767894.823310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767894.823317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767894.823325:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:1.0:1775767894.823331:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823337:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767894.823349:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767894.823352:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076854400:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767894.823370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076854528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767894.823384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767894.823391:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767894.823397:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:1.0:1775767894.823403:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823408:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767894.823416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767894.823420:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767894.823439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076854656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767894.823453:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823458:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767894.823461:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767894.823466:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00100000:2.0:1775767894.823472:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350930700 x1862023076854400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767894.823474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823490:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767894.823498:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076854400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767894.823499:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767894.823520:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.823525:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767894.823530:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076854784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767894.823534:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076854400:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767894.823556:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767894.823564:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767894.823566:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767894.823578:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:1.0:1775767894.823585:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823592:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767894.823598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767894.823602:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767894.823610:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350932a00 x1862023076854528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767894.823625:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076854912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767894.823625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076854528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.823652:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.823655:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.823669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767894.823681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767894.823689:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823694:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767894.823697:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767894.823702:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767894.823716:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00100000:2.0:1775767894.823723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350932300 x1862023076854656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767894.823727:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823743:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767894.823744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076854656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767894.823757:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767894.823790:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076855040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767894.823831:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767894.823834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767894.823843:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767894.823847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767894.823848:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767894.823852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767894.823860:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00100000:2.0:1775767894.823864:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854784:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767894.823869:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767894.823876:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767894.823883:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775767894.823890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767894.823903:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076855168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767894.823904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350930e00 x1862023076854784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767894.823919:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.823928:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767894.823930:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767894.823932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076854784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767894.823947:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:2.0:1775767894.823949:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767894.823958:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767894.823958:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.823966:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854784:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767894.823972:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767894.823982:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767894.823996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767894.824001:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076855296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767894.824043:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767894.824054:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350931880 x1862023076854912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767894.824063:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076854912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.824069:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.824071:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.824080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076854912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767894.824085:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767894.824101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767894.824109:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350930000 x1862023076855040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767894.824117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076855040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.824122:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.824124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.824126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767894.824130:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767894.824142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767894.824147:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350930a80 x1862023076855168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767894.824156:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076855168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.824179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.824181:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.824185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855168:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767894.824190:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767894.824194:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767894.824197:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767894.824198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855296:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767894.824204:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4396 00000100:00080000:1.0:1775767894.824210:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767894.824214:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767894.824217:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767894.824228:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767894.824232:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350931c00 x1862023076855296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767894.824242:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076855296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.824248:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.824251:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767894.824254:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076855424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767894.824257:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767894.824273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767894.824289:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767894.824296:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767894/real 1775767894] req@ffffa04350932680 x1862023076855424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767894.824303:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076855424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767894.824308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767894.824310:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767894.824311:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076855424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767895.015139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767895.015142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767895.847139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767895.847164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767896.039130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767896.039130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767896.871139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767896.871204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767897.063126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767897.063126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767897.895106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767897.895173:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767898.087115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767898.087115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767898.919133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767898.919209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767899.111127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767899.111127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767899.943114:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767899.943120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767899.943125:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767899.943129:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775767899.943136:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767899.943146:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943156:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943163:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767899.943204:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076855552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767899.943237:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943240:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767899.943241:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767899.943245:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943248:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943251:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943259:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767899.943270:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076855680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767899.943278:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076855552:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767899.943281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943286:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767899.943289:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767899.943292:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943294:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943297:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943300:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767899.943309:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076855808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767899.943316:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943318:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767899.943320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767899.943322:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943325:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943328:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943331:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767899.943338:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076855936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767899.943344:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943347:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767899.943348:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767899.943351:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943353:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943357:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943362:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767899.943375:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076856064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767899.943383:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767899.943388:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943392:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767899.943394:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767899.943399:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943404:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943408:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943414:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767899.943421:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767899.943427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076856192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767899.943443:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943446:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767899.943448:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767899.943451:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855936:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767899.943454:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943458:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943461:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943465:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767899.943480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076856320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767899.943489:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767899.943494:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767899.943499:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856064:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767899.943500:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943507:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943512:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767899.943522:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767899.943534:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076856448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767899.943542:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767899.943548:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767899.943553:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767899.943555:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767899.943560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856448:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767899.943562:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4401 00000100:00080000:1.0:1775767899.943570:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767899.943574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767899.943579:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767899.943592:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076856576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767899.943628:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767900.135104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767900.135119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775767900.967193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767900.967201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767901.159101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767901.159126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767901.991121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767901.991203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767902.183111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767902.183111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767903.015180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767903.015303:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775767903.015332:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775767903.015374:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767903.015390:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015436:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015451:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015462:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767903.015471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350930700 x1862023076856576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775767903.015472:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015485:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015498:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767903.015512:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775767903.015512:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076856576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775767903.015526:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:3.0:1775767903.015532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015547:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015553:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767903.015562:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350932a00 x1862023076856448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015576:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076856448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015591:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767903.015598:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350931880 x1862023076856064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015611:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076856064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015620:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015624:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015628:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767903.015634:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350930e00 x1862023076856192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076856192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015675:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015679:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767903.015696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350932300 x1862023076856320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076856320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076856320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767903.015731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350931c00 x1862023076855680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076855680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015755:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015758:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767903.015765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350930a80 x1862023076855808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015776:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076855808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015784:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015788:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767903.015798:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350930000 x1862023076855936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076855936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076855936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767903.015829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767899/real 1775767903] req@ffffa04350932680 x1862023076855552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767903.015842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076855552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767903.015850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767903.015853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767903.015856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076855552:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767903.207133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767903.207133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767904.039138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767904.039197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767904.231109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767904.231117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767905.063144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767905.063187:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063191:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767905.063195:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063203:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4406 00010000:00010000:2.0:1775767905.063205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767905.063212:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063218:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767905.063253:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076856704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063289:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063293:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767905.063295:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063300:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063305:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063310:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767905.063334:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076856832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063352:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063356:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767905.063358:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063362:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063366:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063370:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063375:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767905.063387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076856960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767905.063403:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063406:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063410:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063413:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063418:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767905.063427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076857088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063443:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063446:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767905.063448:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063460:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063464:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063468:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063472:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767905.063483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076857216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063492:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063496:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767905.063497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063501:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063505:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063509:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063514:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767905.063523:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076857344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063532:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767905.063536:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063540:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063544:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063548:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767905.063563:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076857472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063574:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063577:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767905.063579:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063583:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063587:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063591:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063596:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767905.063605:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076857600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767905.063615:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767905.063618:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767905.063620:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767905.063624:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4406 00000100:00080000:1.0:1775767905.063629:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767905.063633:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767905.063638:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767905.063647:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076857728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767905.063762:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076856704:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775767905.063865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350932680 x1862023076856704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064023:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076856704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064068:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076856704:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767905.064077:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767905.064104:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064120:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350930000 x1862023076856832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076856832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064156:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064159:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767905.064165:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767905.064187:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064199:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350930a80 x1862023076856960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064210:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076856960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064220:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064224:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076856960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767905.064230:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767905.064252:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064262:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350931c00 x1862023076857088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064272:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076857088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064285:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064288:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064291:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767905.064297:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767905.064316:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064328:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350932300 x1862023076857216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064339:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076857216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064347:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064350:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767905.064359:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767905.064378:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064388:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350930e00 x1862023076857344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076857344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064409:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064412:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767905.064417:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767905.064434:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350931880 x1862023076857472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064453:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076857472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064460:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064463:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064466:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767905.064471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767905.064487:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350932a00 x1862023076857600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076857600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064514:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064517:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064520:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767905.064526:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767905.064546:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767905.064557:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767905/real 1775767905] req@ffffa04350930700 x1862023076857728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767905.064567:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076857728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767905.064574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767905.064577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767905.064581:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076857728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767905.255110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767905.255113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767906.087141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767906.087377:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767906.279088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767906.279088:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767907.111156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767907.111173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767907.303110:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767907.303111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767908.135117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767908.135148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767908.327143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767908.327143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767909.159108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767909.159164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767909.351141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767909.351143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767910.183127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767910.183132:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767910.183145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183155:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4411 00010000:00010000:3.0:1775767910.183157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767910.183165:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183173:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767910.183211:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076857856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183233:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183236:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767910.183237:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183240:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183244:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183247:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767910.183255:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767910.183261:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076857856:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767910.183267:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076857984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183275:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183278:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767910.183279:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183282:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183284:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183287:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767910.183290:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767910.183297:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076858112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183304:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183306:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767910.183308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183309:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183312:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183316:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767910.183318:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767910.183321:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767910.183329:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076858240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767910.183343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183347:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183351:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183355:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350930700 x1862023076857856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767910.183362:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767910.183379:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076858368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183399:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767910.183403:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767910.183407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076857856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767910.183409:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183414:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183421:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775767910.183422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767910.183426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767910.183428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183433:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076857856:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767910.183439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076858496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183448:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767910.183450:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183452:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183455:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767910.183456:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767910.183461:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767910.183469:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767910.183478:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076858624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767910.183485:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767910.183489:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767910.183491:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767910.183492:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767910.183497:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00080000:1.0:1775767910.183502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183507:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183507:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350932a00 x1862023076857984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767910.183511:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767910.183524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076858752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767910.183526:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076857984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767910.183547:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767910.183549:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767910.183550:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767910.183553:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767910.183554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767910.183559:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4411 00000100:00100000:3.0:1775767910.183560:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076857984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767910.183562:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767910.183567:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858112:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775767910.183572:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767910.183583:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076858880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767910.183591:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183601:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350931880 x1862023076858112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076858112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183630:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183633:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183635:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767910.183640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767910.183654:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350930e00 x1862023076858240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076858240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183679:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183681:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767910.183686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767910.183695:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183700:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350932300 x1862023076858368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076858368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183715:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183717:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076858368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767910.183720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767910.183730:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183735:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350931c00 x1862023076858496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076858496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183753:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183755:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767910.183767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767910.183777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350930a80 x1862023076858624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076858624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183798:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767910.183801:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767910.183810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350930000 x1862023076858752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076858752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183828:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183830:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767910.183834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767910.183843:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767910.183848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767910/real 1775767910] req@ffffa04350932680 x1862023076858880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767910.183854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076858880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767910.183859:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767910.183861:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767910.183863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076858880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767910.375132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767910.375132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767911.207136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767911.207179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767911.399130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767911.399130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767912.231113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767912.231156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767912.423092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767912.423107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767913.255103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767913.255171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767913.447111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767913.447112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767914.279127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767914.279229:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767914.471128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767914.471128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767915.303151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767915.303174:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303182:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767915.303187:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767915.303196:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767915.303198:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303215:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303223:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767915.303259:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076859008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303285:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303288:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767915.303290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303295:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303298:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303305:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767915.303314:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076859136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303323:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303325:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767915.303327:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303330:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303332:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303334:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303338:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767915.303344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076859264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303352:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303356:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767915.303358:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00100000:3.0:1775767915.303364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076859008:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767915.303378:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303387:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303393:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767915.303409:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076859392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303423:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767915.303424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303430:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303434:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303438:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303443:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767915.303454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076859520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303467:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767915.303471:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767915.303472:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767915.303483:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303487:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303491:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303500:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767915.303513:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076859648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303522:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303524:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767915.303526:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303531:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303534:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303539:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767915.303543:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350932680 x1862023076859008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767915.303547:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767915.303578:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076859776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303593:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303596:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767915.303597:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303602:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303605:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767915.303609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076859008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767915.303612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303622:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767915.303636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076859904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767915.303643:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767915.303645:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767915.303649:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767915.303653:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4416 00000100:00080000:1.0:1775767915.303661:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767915.303673:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767915.303683:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775767915.303691:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767915.303697:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076860032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767915.303706:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.303729:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076859008:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775767915.303745:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767915.303773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.303788:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350930000 x1862023076859136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.303802:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076859136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.303812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.303817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.303821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767915.303852:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767915.303873:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.303882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350930a80 x1862023076859264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.303899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076859264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.303905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.303908:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.303911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767915.303919:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767915.303941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.303953:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350931c00 x1862023076859392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.303966:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076859392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.303975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.303979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.303990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767915.303998:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767915.304016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.304024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350932300 x1862023076859520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.304054:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076859520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.304061:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.304063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.304065:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076859520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767915.304070:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767915.304083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.304090:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350930e00 x1862023076859648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.304098:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076859648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.304103:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.304105:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.304107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767915.304111:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767915.304122:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.304135:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350931880 x1862023076859776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.304143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076859776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.304148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.304150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.304153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767915.304156:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767915.304167:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.304173:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350932a00 x1862023076859904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.304179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076859904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.304190:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.304192:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.304194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076859904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767915.304198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767915.304207:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767915.304212:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767915/real 1775767915] req@ffffa04350930700 x1862023076860032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767915.304219:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076860032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767915.304223:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767915.304226:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767915.304228:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767915.495185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767915.495186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767916.327129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767916.327185:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767916.519100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767916.519144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767917.351157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767917.351197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767917.543150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767917.543156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767918.375137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767918.375179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767918.567127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767918.567127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767919.399141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767919.399165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767919.591116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767919.591116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767920.423104:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767920.423110:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423116:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767920.423120:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423128:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423138:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423145:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767920.423153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767920.423180:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076860160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423204:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423207:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767920.423209:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423212:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423214:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423217:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423225:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767920.423234:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076860288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423242:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767920.423246:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076860160:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767920.423247:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767920.423250:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423257:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423260:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423262:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423266:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767920.423279:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076860416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423286:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423289:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767920.423290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423292:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423297:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423299:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423302:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767920.423306:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767920.423313:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076860544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423320:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423323:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767920.423324:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423327:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423330:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423333:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423337:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767920.423341:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350930700 x1862023076860160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767920.423346:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076860672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423365:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423367:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767920.423368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767920.423371:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076860160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767920.423374:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423379:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767920.423385:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423388:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767920.423393:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076860800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767920.423394:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423404:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423409:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076860160:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767920.423410:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423417:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767920.423419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423425:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423430:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767920.423433:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860288:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767920.423438:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423445:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767920.423454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350932a00 x1862023076860288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767920.423471:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076860928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767920.423480:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767920.423481:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076860288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767920.423485:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767920.423488:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423495:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00080000:1.0:1775767920.423501:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767920.423503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767920.423506:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423526:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767920.423536:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076861056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767920.423541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767920.423542:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767920.423548:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767920.423549:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767920.423554:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4422 00000100:00100000:3.0:1775767920.423554:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860288:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767920.423560:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767920.423565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767920.423566:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767920.423573:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767920.423584:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076861184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767920.423597:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423609:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350931880 x1862023076860416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423618:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076860416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423630:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423632:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767920.423636:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767920.423647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423653:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350930e00 x1862023076860544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076860544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423665:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423666:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767920.423671:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767920.423681:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350932300 x1862023076860672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423693:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076860672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423700:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423703:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076860672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767920.423707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767920.423716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350931c00 x1862023076860800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076860800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423735:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423737:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767920.423749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767920.423760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350930a80 x1862023076860928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423772:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076860928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423777:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423779:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423780:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076860928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767920.423784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767920.423793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350930000 x1862023076861056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076861056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423811:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423814:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767920.423817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767920.423826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767920.423833:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767920/real 1775767920] req@ffffa04350932680 x1862023076861184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767920.423844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076861184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767920.423852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767920.423855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767920.423858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767920.615158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767920.615159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767921.447138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767921.447138:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767921.639108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767921.639109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767922.471139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767922.471220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767922.663130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767922.663130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767923.495135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767923.495178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767923.687126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767923.687126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767924.519139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767924.519230:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767924.711122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767924.711122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767925.543150:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.543161:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767925.543167:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.543179:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.543192:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767925.543213:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767925.543213:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767925.544349:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767925.544391:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076861312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767925.544428:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.544434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767925.544438:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.544444:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544454:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.544461:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767925.544478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076861312:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767925.544486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767925.544506:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076861440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767925.544519:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.544523:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767925.544524:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.544528:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544533:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.544539:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.544544:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767925.544555:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076861568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767925.544566:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.544574:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767925.544574:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767925.544583:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.544589:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544595:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.544600:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.544606:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767925.544620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076861696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767925.544630:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.544634:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767925.544636:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.544639:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544644:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767925.544649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350932680 x1862023076861312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767925.544654:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.544667:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767925.544686:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076861824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767925.544694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076861312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.544732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767925.544734:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767925.544739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767925.544745:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775767925.544749:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076861312:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767925.544751:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767925.544766:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544772:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.544776:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.544782:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767925.544785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767925.544801:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076861952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767925.544813:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.544817:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767925.544819:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767925.544825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767925.544831:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00080000:1.0:1775767925.544837:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.544843:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.544848:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767925.544853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350930000 x1862023076861440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767925.544865:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076862080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767925.544866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076861440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767925.544944:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767925.544944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767925.544956:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767925.544956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767925.544961:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767925.544962:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861440:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767925.544978:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00100000:3.0:1775767925.544978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767925.544994:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.545001:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.545008:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767925.545014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767925.545055:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076862208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767925.545055:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350930a80 x1862023076861568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076861568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767925.545123:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767925.545142:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767925.545146:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767925.545148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767925.545152:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545158:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767925.545164:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4427 00000100:00100000:3.0:1775767925.545169:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861696:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767925.545170:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767925.545181:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767925.545189:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775767925.545200:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767925.545205:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076862336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767925.545212:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350931c00 x1862023076861696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545242:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076861696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545251:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545255:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545257:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767925.545262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767925.545275:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767925.545281:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350932300 x1862023076861824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545288:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076861824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545299:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545302:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545304:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076861824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767925.545320:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767925.545331:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767925.545339:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350930e00 x1862023076861952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076861952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545368:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076861952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767925.545375:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767925.545397:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767925.545407:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350931880 x1862023076862080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545416:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076862080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545421:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545424:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545427:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767925.545431:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767925.545443:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767925.545449:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350932a00 x1862023076862208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545456:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076862208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545461:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545478:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767925.545485:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767925.545500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767925.545508:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767925/real 1775767925] req@ffffa04350930700 x1862023076862336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767925.545516:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076862336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767925.545521:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767925.545524:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767925.545526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076862336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767925.735085:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767925.735150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767926.567152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767926.567233:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767926.759123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767926.759123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767927.591203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767927.591215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767927.783111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767927.783112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767928.615139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767928.615170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767928.807098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767928.807107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767929.639117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767929.639185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767929.831118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767929.831118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775767930.663123:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775767930.663144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767930.663145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767930.663145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767930.663159:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663170:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663178:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767930.663214:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076862464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767930.663239:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767930.663245:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663250:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663254:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663259:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663267:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076862464:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775767930.663270:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767930.663288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076862592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767930.663301:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663305:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767930.663307:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663311:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663323:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775767930.663324:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775767930.663331:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767930.663344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076862720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767930.663361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350930700 x1862023076862464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767930.663364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663372:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767930.663376:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663384:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663388:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663394:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663394:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076862464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767930.663402:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767930.663434:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076862848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767930.663441:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767930.663442:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767930.663443:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767930.663447:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767930.663451:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767930.663452:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663457:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767930.663458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076862464:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767930.663465:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767930.663471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775767930.663482:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767930.663486:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076862976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767930.663500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663504:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767930.663507:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767930.663507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767930.663516:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00100000:3.0:1775767930.663522:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350932a00 x1862023076862592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767930.663523:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663533:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663538:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076862592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767930.663539:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775767930.663551:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767930.663558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076863104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767930.663559:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767930.663569:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767930.663579:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663583:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767930.663585:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663591:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4432 00000800:00000100:3.0:1775767930.663592:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767930.663599:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767930.663605:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663606:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350931880 x1862023076862720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767930.663612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767930.663620:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076862720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767930.663628:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076863232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767930.663629:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663635:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767930.663652:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767930.663653:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862848:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767930.663661:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767930.663664:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663669:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663673:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767930.663676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767930.663681:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767930.663686:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767930.663687:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350930e00 x1862023076862848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767930.663698:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076862848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663703:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767930.663711:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076863360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767930.663723:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862976:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775767930.663739:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767930.663741:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767930.663742:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767930.663746:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4432 00000100:00080000:1.0:1775767930.663750:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767930.663754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767930.663757:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767930.663763:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775767930.663767:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350932300 x1862023076862976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767930.663774:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076863488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767930.663801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076862976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663811:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076862976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767930.663819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767930.663831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350931c00 x1862023076863104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767930.663844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076863104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663849:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663851:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767930.663856:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775767930.663865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350930a80 x1862023076863232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767930.663877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076863232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663890:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775767930.663898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767930.663907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350930000 x1862023076863360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767930.663919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076863360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663927:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767930.663930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775767930.663939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767930.663944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767930/real 1775767930] req@ffffa04350932680 x1862023076863488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767930.663951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076863488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767930.663956:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767930.663957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767930.663960:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076863488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767930.855116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767930.855116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767931.687110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767931.687244:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767931.879137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767931.879137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767932.711113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767932.711125:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767932.903119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767932.903119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767933.735115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767933.735158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767933.927152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767933.927152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767934.759115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767934.759132:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767934.951103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767934.951118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767935.783114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775767935.783146:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767935.783158:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767935.783168:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783178:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767935.783186:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775767935.783223:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076863616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767935.783258:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783273:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767935.783275:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767935.783280:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783286:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767935.783291:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767935.783301:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076863616:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775767935.783308:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767935.783327:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076863744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767935.783345:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783350:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767935.783352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775767935.783357:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767935.783406:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825ea00 x1862023076863616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767935.783421:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076863616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767935.783428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767935.783432:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767935.783435:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076863616:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775767935.783445:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783452:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767935.783455:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076863744:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767935.783459:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775767935.783476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767935.783484:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825ed80 x1862023076863744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767935.783493:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076863744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767935.783499:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767935.783501:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767935.783503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076863744:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767935.783586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767935.783641:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076863872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767935.783672:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783676:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767935.783679:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767935.783683:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00100000:0.0:1775767935.783687:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076863872:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767935.783690:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767935.783697:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767935.783704:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767935.783716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767935.783720:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076864000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767935.783731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825c000 x1862023076863872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767935.783734:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783744:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767935.783747:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767935.783750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076863872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767935.783756:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783765:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767935.783769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767935.783774:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767935.783776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767935.783782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076863872:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775767935.783785:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767935.783795:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767935.783801:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076864128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767935.783813:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783816:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767935.783818:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767935.783823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767935.783827:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783833:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767935.783835:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825e680 x1862023076864000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767935.783841:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767935.783851:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767935.783853:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076864000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767935.783862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767935.783865:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775767935.783869:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076864256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767935.783871:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767935.783886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864128:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767935.783887:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783893:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767935.783895:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767935.783900:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4437 00000800:00000100:0.0:1775767935.783904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775767935.783908:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767935.783914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825f100 x1862023076864128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767935.783916:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767935.783927:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767935.783933:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076864128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767935.783941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076864384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767935.783946:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767935.783956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767935.783965:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864128:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775767935.783978:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767935.783982:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767935.783984:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767935.783985:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767935.783993:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:3.0:1775767935.783998:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767935.784003:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:0.0:1775767935.784009:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:3.0:1775767935.784009:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767935.784020:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825e300 x1862023076864256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767935.784047:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076864512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767935.784062:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767935.784066:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076864256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767935.784070:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767935.784075:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767935.784080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767935.784084:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4437 00000100:00080000:0.0:1775767935.784088:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775767935.784092:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767935.784093:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767935.784103:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864384:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775767935.784104:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775767935.784113:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767935.784123:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775767935.784130:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076864640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767935.784134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825d500 x1862023076864384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767935.784162:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076864384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767935.784169:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767935.784171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767935.784173:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767935.784186:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767935.784200:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767935.784207:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825dc00 x1862023076864512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767935.784215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076864512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767935.784222:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767935.784225:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767935.784228:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864512:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775767935.784228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767935.784235:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767935.784244:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767935.784249:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767935/real 1775767935] req@ffffa0435825d880 x1862023076864640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767935.784256:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076864640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767935.784261:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767935.784263:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767935.784264:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076864640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767935.975128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767935.975128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767936.807108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767936.807134:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767936.999109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767936.999110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767937.831143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767937.831199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767938.023119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767938.023119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767938.855111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767938.855164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767939.047113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767939.047113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767939.879134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767939.879189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767940.071120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767940.071126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767940.903116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767940.903135:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767940.903145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903153:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4442 00010000:00010000:3.0:1775767940.903157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767940.903162:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903168:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767940.903204:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076864768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767940.903226:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903228:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767940.903230:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903233:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903235:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903238:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903246:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767940.903255:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076864896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767940.903262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076864768:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767940.903266:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903271:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767940.903273:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903276:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903278:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903283:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767940.903291:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076865024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767940.903298:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903300:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767940.903301:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903303:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903350:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903355:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767940.903358:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767940.903367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076865152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767940.903375:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903378:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767940.903379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903382:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903385:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903387:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903391:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767940.903401:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076865280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767940.903404:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350932680 x1862023076864768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767940.903416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903423:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767940.903427:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903432:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903435:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903437:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903440:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767940.903442:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076864768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767940.903449:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076865408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767940.903460:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903471:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767940.903472:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767940.903477:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767940.903481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076864768:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767940.903481:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903486:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903491:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767940.903503:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076865536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767940.903510:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767940.903513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864896:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767940.903514:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767940.903517:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903524:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903526:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903529:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767940.903532:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767940.903540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076865664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767940.903547:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767940.903548:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767940.903553:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767940.903555:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767940.903559:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4442 00000100:00080000:1.0:1775767940.903562:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767940.903565:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903569:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350930000 x1862023076864896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767940.903570:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767940.903582:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076865792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767940.903589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076864896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903599:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076864896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767940.903612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767940.903629:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350930a80 x1862023076865024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903649:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076865024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903662:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903665:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767940.903673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767940.903689:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350931c00 x1862023076865152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076865152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903715:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767940.903726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767940.903743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350932300 x1862023076865280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076865280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076865280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767940.903791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767940.903808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350930e00 x1862023076865408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076865408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903840:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767940.903863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767940.903885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350931880 x1862023076865536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076865536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767940.903930:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767940.903947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.903957:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350932a00 x1862023076865664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.903968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076865664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.903976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.903980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.903983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767940.903988:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767940.904006:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767940.904019:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767940/real 1775767940] req@ffffa04350930700 x1862023076865792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767940.904057:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076865792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767940.904065:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767940.904069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767940.904072:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076865792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767941.095111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767941.095111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767941.927167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767941.927238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767942.119121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767942.119121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767942.951129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767942.951200:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767943.143086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767943.143101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767943.975117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767943.975189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767944.167126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767944.167127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767944.999114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767944.999140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767945.191093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767945.191097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775767946.023121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767946.023130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775767946.023130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767946.023133:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023144:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023153:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023160:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767946.023182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767946.023196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076865920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767946.023216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767946.023219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767946.023220:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023223:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023226:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023228:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023237:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767946.023246:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076866048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767946.023254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767946.023258:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076865920:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767946.023258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767946.023260:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023281:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023285:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023288:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023292:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767946.023303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076866176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767946.023310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767946.023313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767946.023314:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023317:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023328:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023330:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023334:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767946.023339:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767946.023345:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076866304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767946.023353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767946.023356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767946.023358:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023361:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023363:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023366:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767946.023378:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350930700 x1862023076865920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767946.023384:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076866432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767946.023397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767946.023398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076865920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767946.023407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767946.023411:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767946.023413:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023419:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775767946.023419:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00100000:0.0:1775767946.023426:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076865920:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775767946.023428:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023435:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023441:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767946.023446:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767946.023457:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076866560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767946.023466:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767946.023474:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767946.023478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767946.023480:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350932a00 x1862023076866048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767946.023482:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023491:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023498:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023504:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023506:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076866048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767946.023511:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775767946.023522:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767946.023527:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076866688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767946.023528:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023537:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767946.023541:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767946.023545:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767946.023547:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767946.023550:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775767946.023553:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023559:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767946.023564:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775767946.023570:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767946.023582:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076866816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767946.023591:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775767946.023599:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767946.023604:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350931880 x1862023076866176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767946.023605:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767946.023609:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767946.023619:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4447 00000100:00080000:1.0:1775767946.023628:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767946.023632:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076866176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767946.023636:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767946.023644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775767946.023647:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775767946.023648:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767946.023658:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767946.023665:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076866944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767946.023674:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350930e00 x1862023076866304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076866304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023705:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767946.023712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767946.023723:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350932300 x1862023076866432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023735:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076866432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076866432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767946.023771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767946.023782:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350931c00 x1862023076866560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076866560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023804:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023805:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767946.023809:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767946.023818:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350930a80 x1862023076866688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023836:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076866688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023841:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023843:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767946.023849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767946.023859:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350930000 x1862023076866816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023871:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076866816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767946.023883:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767946.023898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767946.023907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767946/real 1775767946] req@ffffa04350932680 x1862023076866944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767946.023919:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076866944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767946.023927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767946.023930:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767946.023934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076866944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767946.215098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767946.215098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767947.047114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767947.047189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767947.239095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767947.239114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767948.071176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767948.071233:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767948.263195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767948.263195:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767949.095111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767949.095195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767949.287158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767949.287158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767950.119131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767950.119186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767950.311105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767950.311139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775767951.143158:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767951.143161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767951.143170:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767951.143175:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143187:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143199:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775767951.143206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767951.143207:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767951.143264:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076867072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143292:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143296:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767951.143298:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143303:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143308:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143314:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767951.143330:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076867072:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767951.143345:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076867200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143360:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143363:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767951.143365:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143370:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143374:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143394:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143402:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767951.143415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767951.143420:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076867328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143433:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143439:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767951.143441:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143446:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143453:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143457:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143462:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767951.143470:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7c700 x1862023076867072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767951.143476:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076867456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143537:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767951.143541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143547:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143551:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767951.143554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076867072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767951.143559:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143571:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775767951.143580:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.143592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767951.143593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076867584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767951.143609:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076867072:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767951.143622:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143629:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767951.143632:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143639:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143642:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143646:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767951.143651:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867200:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767951.143652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767951.143676:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076867712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143686:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767951.143692:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767951.143697:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767951.143700:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143707:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143710:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143716:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767951.143717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7fb80 x1862023076867200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767951.143727:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767951.143751:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076867200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767951.143752:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076867840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143823:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767951.143824:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767951.143833:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767951.143836:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767951.143841:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767951.143847:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143854:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767951.143860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767951.143865:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143873:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767951.143879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767951.143887:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076867968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767951.143905:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767951.143908:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767951.143910:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767951.143915:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4452 00000100:00080000:2.0:1775767951.143919:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767951.143922:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767951.143926:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767951.143928:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767951.143947:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076868096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767951.143954:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7ce00 x1862023076867328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.143981:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076867328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.143987:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.143991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.143993:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767951.143999:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767951.144014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144020:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7ed80 x1862023076867456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144049:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076867456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144055:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144057:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144075:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767951.144081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767951.144099:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7f800 x1862023076867584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076867584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144133:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144135:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076867584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767951.144158:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767951.144171:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144179:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7ea00 x1862023076867712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144189:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076867712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144195:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767951.144205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767951.144239:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144251:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7df80 x1862023076867840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144276:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076867840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144291:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767951.144295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767951.144308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144314:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7d880 x1862023076867968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144321:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076867968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144327:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144329:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144330:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076867968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767951.144334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767951.144343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767951.144349:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767951/real 1775767951] req@ffffa0434aa7c700 x1862023076868096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767951.144356:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076868096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767951.144361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767951.144363:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767951.144366:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767951.335176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767951.335176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767952.167147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767952.167238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767952.359144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767952.359144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767953.191151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767953.191170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767953.383152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767953.383155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767954.215119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767954.215206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767954.407120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767954.407151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767955.239163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767955.239220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767955.431118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767955.431159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767956.263162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767956.263177:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263189:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767956.263196:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263207:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263217:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263225:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775767956.263251:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767956.263263:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076868224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263288:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263291:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767956.263306:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263311:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263320:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767956.263329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076868224:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767956.263340:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076868352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767956.263353:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263356:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263359:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263363:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263366:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767956.263374:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076868480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263380:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767956.263384:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263386:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263388:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263390:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263395:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767956.263407:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076868608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263414:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263416:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767956.263417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263420:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263423:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263426:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263428:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775767956.263434:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767956.263443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076868736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263452:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263455:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767956.263456:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263460:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263464:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263467:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767956.263479:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076868864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263485:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263488:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767956.263489:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7e680 x1862023076868224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767956.263491:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263512:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263517:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767956.263519:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076868224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767956.263532:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775767956.263539:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767956.263542:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775767956.263547:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263554:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076868224:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767956.263569:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076868992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767956.263580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767956.263582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767956.263587:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868352:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767956.263587:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263599:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263605:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767956.263612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767956.263621:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076869120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767956.263628:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767956.263636:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767956.263638:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767956.263641:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767956.263648:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4457 00000100:00080000:2.0:1775767956.263651:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767956.263655:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263659:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7e300 x1862023076868352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767956.263664:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767956.263686:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076869248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767956.263688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076868352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263696:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767956.263732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767956.263759:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7c000 x1862023076868480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.263778:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076868480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263785:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263788:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263791:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767956.263796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767956.263808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7f100 x1862023076868608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.263822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076868608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263828:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263830:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767956.263836:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767956.263847:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7c380 x1862023076868736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.263862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076868736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263870:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263872:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076868736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767956.263876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767956.263887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7dc00 x1862023076868864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.263901:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076868864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263908:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263910:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767956.263915:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767956.263926:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.263932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7d500 x1862023076868992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.263939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076868992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.263950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.263954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.263957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076868992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767956.263986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076869120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767956.264010:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.264024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7f480 x1862023076869120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.264073:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076869120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.264084:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.264088:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.264093:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076869120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767956.264101:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076869248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767956.264126:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767956.264140:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767956/real 1775767956] req@ffffa0434aa7e680 x1862023076869248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767956.264154:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076869248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767956.264163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767956.264165:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767956.264185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076869248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767956.455120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767956.455123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775767957.287204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767957.287210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767957.479168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767957.479177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767958.311171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767958.311233:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767958.503105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767958.503175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775767959.335211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767959.335216:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775767959.527198:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767959.527198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767960.359153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767960.359206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767960.551152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767960.551182:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767961.383189:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767961.383203:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383211:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767961.383215:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383228:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383244:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383254:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775767961.383258:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767961.383304:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076869376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767961.383330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383334:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767961.383335:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383339:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383341:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383344:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767961.383368:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076869504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767961.383379:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076869376:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767961.383385:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383392:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767961.383393:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383396:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383427:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383431:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383437:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767961.383447:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076869632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767961.383460:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767961.383462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383465:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767961.383467:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383472:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383477:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383479:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767961.383492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076869760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767961.383499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767961.383504:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383507:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383509:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383514:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767961.383515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7c700 x1862023076869376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767961.383524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767961.383541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076869888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767961.383556:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076869376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767961.383568:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767961.383585:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767961.383591:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767961.383594:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383602:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767961.383603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.383611:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076869376:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767961.383615:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383627:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767961.383643:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076870016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767961.383646:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869504:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767961.383676:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383680:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767961.383682:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383686:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383693:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767961.383700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767961.383705:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383710:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767961.383716:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7d880 x1862023076869504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767961.383723:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076870144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767961.383765:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767961.383768:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076869504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767961.383777:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767961.383782:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383800:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:0.0:1775767961.383804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767961.383815:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767961.383817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767961.383823:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767961.383824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869504:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767961.383834:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767961.383842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767961.383848:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076870272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767961.383862:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767961.383864:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767961.383867:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767961.383871:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4463 00000100:00080000:2.0:1775767961.383875:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767961.383877:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767961.383882:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775767961.383889:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767961.383893:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076870400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767961.383909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7df80 x1862023076869632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.383922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076869632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.383930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.383933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.383935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767961.383940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767961.383955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.383961:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7ea00 x1862023076869760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.383968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076869760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.383973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.383976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.383978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767961.383982:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767961.383992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.383997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7f800 x1862023076869888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.384009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076869888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.384014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.384016:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.384018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076869888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767961.384055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767961.384067:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.384074:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7ed80 x1862023076870016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.384082:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076870016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.384088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.384090:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.384092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767961.384096:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775767961.384105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.384111:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7ce00 x1862023076870144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.384132:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076870144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.384142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.384144:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.384146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767961.384150:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767961.384164:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.384171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7fb80 x1862023076870272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.384178:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076870272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.384183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.384185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.384187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767961.384190:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775767961.384200:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767961.384205:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767961/real 1775767961] req@ffffa0434aa7c700 x1862023076870400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767961.384212:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076870400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767961.384217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767961.384218:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775767961.384221:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076870400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767961.575131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767961.575144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767962.407124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767962.407201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767962.599137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767962.599147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767963.431121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767963.431148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767963.623143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767963.623144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767964.455144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767964.455193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767964.647114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767964.647115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767965.479140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767965.479175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767965.671135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767965.671135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767966.503138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767966.503160:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503166:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767966.503169:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503187:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503196:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767966.503202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767966.503234:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076870528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503266:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503269:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767966.503271:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503274:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503278:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503281:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503289:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767966.503300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076870656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503309:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767966.503309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076870528:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767966.503316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767966.503318:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503321:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503324:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503327:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767966.503340:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076870784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503348:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767966.503350:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503352:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503357:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503359:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503363:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767966.503369:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076870912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503375:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503378:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767966.503379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503381:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503383:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503386:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503389:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767966.503397:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076871040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767966.503400:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767966.503414:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503417:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767966.503419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503423:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503428:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503432:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767966.503436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767966.503452:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076871168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767966.503463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870912:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775767966.503464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767966.503468:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503472:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503476:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503479:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503483:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767966.503489:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775767966.503494:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076871296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767966.503507:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503512:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00100000:1.0:1775767966.503513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871168:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767966.503518:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503522:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503527:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767966.503536:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076871424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767966.503543:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767966.503547:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775767966.503548:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871296:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767966.503549:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767966.503558:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4468 00000100:00080000:0.0:1775767966.503561:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767966.503563:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767966.503566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767966.503576:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076871552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767966.503591:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767966.503611:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767966.695120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767966.695142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767967.527170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767967.527241:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767967.719149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767967.719175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767968.551143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767968.551325:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767968.743202:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767968.743206:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767969.575161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775767969.575332:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775767969.575377:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775767969.575426:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767969.575443:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575497:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575510:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575518:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575528:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575538:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775767969.575549:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349642a00 x1862023076871552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775767969.575550:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775767969.575589:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076871552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775767969.575596:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775767969.575608:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00080000:1.0:1775767969.575617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575625:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575631:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767969.575641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349642300 x1862023076871296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575656:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076871296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575670:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575673:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767969.575683:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349643100 x1862023076871040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076871040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575705:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575723:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767969.575734:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349640380 x1862023076871168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076871168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767969.575775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349643b80 x1862023076870656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575799:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076870656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767969.575832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349643800 x1862023076870784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076870784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575853:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575857:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767969.575866:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349640a80 x1862023076870912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076870912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575890:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575893:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076870912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767969.575901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349641180 x1862023076870528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.575913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076870528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.575921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.575924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.575928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076870528:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775767969.575991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767966/real 1775767969] req@ffffa04349640700 x1862023076871424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767969.576005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076871424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767969.576014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767969.576019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767969.576022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076871424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767969.767150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767969.767150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767970.599145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767970.599252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767970.791155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767970.791164:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767971.623131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767971.623151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767971.623164:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623175:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623194:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767971.623228:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076871680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623260:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623265:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767971.623268:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623273:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623279:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623284:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623294:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1775767971.623299:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767971.623313:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076871808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767971.623341:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767971.623341:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076871680:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767971.623356:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623361:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623369:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623378:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767971.623395:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076871936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623409:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623412:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767971.623413:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623416:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623421:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623425:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623429:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767971.623441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076872064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767971.623456:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767971.623468:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623473:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767971.623476:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623490:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623497:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623503:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767971.623531:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349641500 x1862023076871680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767971.623533:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076872192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623593:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767971.623600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076871680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767971.623601:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767971.623608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623615:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623620:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623637:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767971.623645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767971.623647:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767971.623658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.623665:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076871680:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767971.623673:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076872320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623685:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623688:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767971.623691:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623696:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623701:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623706:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767971.623712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871808:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775767971.623713:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767971.623740:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076872448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767971.623753:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767971.623758:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767971.623764:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1775767971.623765:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767971.623778:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623784:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767971.623795:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349641880 x1862023076871808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767971.623796:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623817:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767971.623826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076871808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.623840:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767971.623842:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076872576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767971.623853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.623863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871808:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767971.623885:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767971.623886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871936:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767971.623891:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767971.623895:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767971.623902:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4473 00000100:00080000:0.0:1775767971.623911:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767971.623919:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767971.623926:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767971.623941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076872704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775767971.623957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.623981:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349640e00 x1862023076871936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.623997:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076871936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624082:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624087:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076871936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767971.624098:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767971.624128:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624144:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349642680 x1862023076872064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624155:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076872064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624164:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624167:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624170:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767971.624177:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767971.624198:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624209:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349641c00 x1862023076872192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624238:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076872192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624248:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624252:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624255:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767971.624263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767971.624288:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624296:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349640700 x1862023076872320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624307:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076872320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624317:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624320:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624324:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767971.624353:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767971.624376:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624387:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349641180 x1862023076872448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076872448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624420:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624424:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767971.624436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767971.624461:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624473:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349640a80 x1862023076872576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624485:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076872576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624494:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624501:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767971.624508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767971.624529:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767971.624540:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767971/real 1775767971] req@ffffa04349643800 x1862023076872704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767971.624551:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076872704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767971.624561:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767971.624564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767971.624575:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076872704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767971.815128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767971.815144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767972.647167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767972.647297:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767972.839166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767972.839172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767973.671135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767973.671200:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767973.863099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767973.863099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767974.695135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767974.695163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767974.887123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767974.887123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767975.719122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767975.719166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767975.911119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767975.911119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767976.743183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767976.743214:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743221:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767976.743226:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743236:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4478 00010000:00010000:1.0:1775767976.743250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767976.743251:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743260:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767976.743309:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076872832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767976.743336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743339:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767976.743341:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743344:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743347:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743352:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743363:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767976.743374:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076872960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767976.743386:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076872832:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767976.743388:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767976.743406:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743410:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743412:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743414:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743417:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767976.743425:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076873088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767976.743432:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767976.743435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743438:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743440:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743443:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743446:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767976.743454:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076873216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767976.743461:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767976.743464:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743467:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743477:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743480:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:2.0:1775767976.743490:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743493:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767976.743509:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076873344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767976.743516:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743519:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767976.743520:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743526:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743529:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743533:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743536:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767976.743541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349643b80 x1862023076872832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767976.743545:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076873472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767976.743577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743593:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767976.743601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743607:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743612:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743616:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767976.743622:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076872832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767976.743625:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767976.743644:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076873600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767976.743646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767976.743663:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767976.743664:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767976.743669:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767976.743672:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767976.743678:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076872832:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767976.743681:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743691:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743698:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743702:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767976.743713:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076873728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767976.743719:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872960:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775767976.743734:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767976.743736:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767976.743737:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767976.743742:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4478 00000100:00080000:0.0:1775767976.743746:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767976.743750:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767976.743754:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767976.743761:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767976.743766:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076873856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767976.743780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349640380 x1862023076872960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.743792:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076872960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.743802:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.743805:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.743810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076872960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767976.743818:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767976.743858:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.743870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349643100 x1862023076873088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.743884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076873088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.743893:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.743896:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.743900:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767976.743906:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767976.743922:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.743930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349642300 x1862023076873216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.743937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076873216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.743943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.743945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.743947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767976.743951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767976.743962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.743967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349642a00 x1862023076873344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.744001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076873344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.744006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.744008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.744011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076873344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767976.744015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767976.744052:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.744061:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349643800 x1862023076873472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.744069:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076873472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.744074:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.744077:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.744079:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767976.744082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767976.744094:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.744099:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349640a80 x1862023076873600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.744106:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076873600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.744121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.744124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.744126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767976.744144:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767976.744158:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.744165:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349641180 x1862023076873728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.744172:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076873728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.744177:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.744178:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.744180:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767976.744184:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767976.744193:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767976.744199:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767976/real 1775767976] req@ffffa04349640700 x1862023076873856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767976.744206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076873856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767976.744210:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767976.744212:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767976.744218:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076873856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767976.935154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767976.935154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767977.767135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767977.767202:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767977.959198:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767977.959199:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767978.791126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767978.791167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767978.983123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767978.983123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767979.815107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767979.815195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767980.007126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767980.007222:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767980.839114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767980.839146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767981.031125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767981.031125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767981.863154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767981.863166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767981.863179:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863190:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863200:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863209:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775767981.863250:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767981.863252:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076873984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767981.863278:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863281:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863283:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863286:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863295:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767981.863306:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076874112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767981.863306:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076873984:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767981.863319:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767981.863322:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863325:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863328:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863330:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863333:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767981.863342:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076874240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863348:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863350:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767981.863351:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863353:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863362:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863365:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863368:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1775767981.863374:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775767981.863378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076874368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863385:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767981.863389:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863392:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863397:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863399:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863401:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767981.863408:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076874496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767981.863418:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863423:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00100000:2.0:1775767981.863425:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349641c00 x1862023076873984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767981.863426:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863431:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863434:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767981.863442:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076874624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863460:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863464:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1775767981.863465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076873984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767981.863467:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863475:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863479:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863483:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:2.0:1775767981.863483:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775767981.863485:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775767981.863490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076873984:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775767981.863496:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076874752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863506:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767981.863509:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863512:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863514:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767981.863516:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863520:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767981.863523:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775767981.863528:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076874880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767981.863536:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767981.863538:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767981.863539:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767981.863542:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4483 00000100:00080000:0.0:1775767981.863545:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1775767981.863549:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767981.863551:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767981.863556:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767981.863565:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349642680 x1862023076874112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767981.863566:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076875008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767981.863606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076874112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863617:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863624:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767981.863631:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767981.863671:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.863679:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349640e00 x1862023076874240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.863688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076874240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767981.863702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767981.863718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.863728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349641880 x1862023076874368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.863740:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076874368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767981.863763:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767981.863780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.863789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349641500 x1862023076874496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.863818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076874496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863828:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863831:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076874496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767981.863840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767981.863860:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.863870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349640700 x1862023076874624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.863882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076874624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863890:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863896:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767981.863911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767981.863947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.863958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349641180 x1862023076874752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.863967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076874752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.863981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.863984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.863986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767981.864003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767981.864018:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.864024:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349640a80 x1862023076874880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.864066:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076874880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.864072:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.864076:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.864078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076874880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767981.864083:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767981.864101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767981.864112:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767981/real 1775767981] req@ffffa04349643800 x1862023076875008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767981.864125:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076875008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767981.864134:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767981.864143:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767981.864154:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767982.055118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767982.055118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767982.887125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767982.887169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767983.079117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767983.079117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767983.911147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775767983.911198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767984.103104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767984.103111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767984.935115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767984.935163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767985.127120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767985.127124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767985.959148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767985.959189:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767986.151100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767986.151100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775767986.983113:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983121:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767986.983125:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775767986.983135:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767986.983136:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00100000:1.0:1775767986.983140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775767986.983147:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983154:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775767986.983187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076875136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983207:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983209:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767986.983211:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983214:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983216:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983219:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983226:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767986.983235:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076875264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767986.983247:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983252:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00100000:2.0:1775767986.983253:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076875136:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767986.983254:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983259:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983265:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767986.983273:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076875392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983280:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983282:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767986.983283:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983286:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983288:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983290:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983293:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767986.983299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076875520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983305:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983307:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767986.983308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983311:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983314:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983316:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775767986.983328:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076875648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983335:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775767986.983337:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775767986.983338:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767986.983342:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983345:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983348:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983351:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983354:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767986.983361:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076875776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983368:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767986.983372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983375:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983377:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983379:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767986.983385:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349642a00 x1862023076875136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767986.983395:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076875904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767986.983407:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767986.983409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767986.983411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00100000:2.0:1775767986.983416:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076875136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767986.983418:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983424:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983430:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775767986.983441:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767986.983443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076876032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767986.983450:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775767986.983454:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767986.983458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076875136:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775767986.983459:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767986.983461:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767986.983468:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4488 00000100:00080000:0.0:1775767986.983470:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767986.983472:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775767986.983475:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775767986.983481:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076876160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767986.983489:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767986.983516:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983526:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349642300 x1862023076875264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983535:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076875264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983541:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983544:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767986.983550:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767986.983601:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983614:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349643100 x1862023076875392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983627:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076875392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983633:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983635:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983637:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767986.983642:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767986.983657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983665:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349640380 x1862023076875520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983672:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076875520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983679:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983681:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767986.983684:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767986.983694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983699:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349643b80 x1862023076875648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983717:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076875648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983722:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983724:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076875648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767986.983730:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767986.983742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349643800 x1862023076875776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076875776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767986.983768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775767986.983778:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983783:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349640a80 x1862023076875904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076875904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983806:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983809:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076875904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767986.983815:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767986.983837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349641180 x1862023076876032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076876032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983890:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775767986.983919:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775767986.983939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767986.983950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767986/real 1775767986] req@ffffa04349640700 x1862023076876160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767986.983963:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076876160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767986.983971:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767986.983974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775767986.983986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775767987.175126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767987.175127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775767988.007147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767988.007210:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767988.199187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767988.199187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767989.031162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767989.031214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767989.223138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767989.223138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767990.055131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767990.055191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767990.247139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767990.247139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775767991.079154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767991.079155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775767991.271102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767991.271103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767992.103113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767992.103140:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767992.103156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767992.103170:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103181:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103193:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103199:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767992.103247:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076876288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767992.103284:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103288:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103291:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103294:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767992.103305:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767992.103323:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076876288:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775767992.103323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076876416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767992.103359:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103365:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103368:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103391:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767992.103396:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775767992.103401:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767992.103410:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076876544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103422:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767992.103424:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103427:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103431:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103436:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767992.103440:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767992.103446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7e680 x1862023076876288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767992.103450:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076876672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103487:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103499:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767992.103503:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103507:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103509:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103513:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767992.103518:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775767992.103526:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076876288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767992.103527:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076876800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103591:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103602:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767992.103604:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767992.103605:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767992.103621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767992.103626:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00100000:1.0:1775767992.103627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076876288:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767992.103632:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103638:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767992.103643:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767992.103656:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767992.103659:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076876928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767992.103685:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103689:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767992.103692:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103697:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4493 00000800:00000100:1.0:1775767992.103705:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767992.103710:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103718:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767992.103724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7f480 x1862023076876416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767992.103731:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767992.103738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076876416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767992.103749:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076877056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767992.103750:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.103766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775767992.103767:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767992.103775:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775767992.103779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767992.103779:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103786:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:2.0:1775767992.103795:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103800:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775767992.103805:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876544:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775767992.103818:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775767992.103859:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775767992.103865:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076877184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767992.103872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7d500 x1862023076876544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767992.103913:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767992.103918:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076876544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767992.103929:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767992.103937:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767992.103950:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4493 00000100:00080000:1.0:1775767992.103954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767992.103956:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767992.103963:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775767992.103970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775767992.103974:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775767992.103980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775767992.103991:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076877312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767992.103994:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767992.104084:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104107:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7dc00 x1862023076876672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076876672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104132:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104136:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104140:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767992.104147:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767992.104170:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104181:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7c380 x1862023076876800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104190:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076876800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104197:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104199:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076876800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767992.104218:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767992.104232:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104239:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7f100 x1862023076876928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104247:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076876928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104252:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104254:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104256:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076876928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767992.104260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767992.104269:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104274:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7c000 x1862023076877056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104281:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076877056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104291:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104293:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104295:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767992.104299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767992.104308:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104313:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7e300 x1862023076877184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104319:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076877184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104324:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104328:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767992.104332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767992.104341:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767992.104346:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767992/real 1775767992] req@ffffa0434aa7e680 x1862023076877312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767992.104352:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076877312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767992.104357:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767992.104359:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767992.104362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076877312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767992.295132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767992.295132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767993.127130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767993.127175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767993.319101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767993.319146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767994.151154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767994.151184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767994.343147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767994.343147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767995.175135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767995.175157:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767995.367086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775767995.367094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775767996.199130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767996.199165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767996.391138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767996.391138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767997.223102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775767997.223125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767997.223137:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223147:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223156:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223163:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775767997.223196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076877440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767997.223243:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223247:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4498 00010000:00010000:3.0:1775767997.223274:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767997.223280:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223284:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223292:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767997.223303:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076877568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767997.223315:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223320:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223323:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223331:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767997.223338:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076877696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223345:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223347:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767997.223348:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223351:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223354:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223357:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223360:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767997.223368:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076877824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767997.223371:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076877440:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767997.223386:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223389:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767997.223390:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223393:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223396:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223399:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223403:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775767997.223412:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076877952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767997.223422:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223424:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223427:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223429:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767997.223437:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076878080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775767997.223451:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775767997.223452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767997.223456:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223461:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00080000:2.0:1775767997.223474:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223479:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223485:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767997.223496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7c700 x1862023076877440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767997.223500:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076878208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767997.223526:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223529:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767997.223531:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223535:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00100000:1.0:1775767997.223536:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076877440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767997.223538:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223544:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223552:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775767997.223558:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775767997.223563:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076878336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767997.223565:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223574:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076877440:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775767997.223575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767997.223583:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767997.223584:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767997.223588:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4498 00000100:00100000:1.0:1775767997.223589:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775767997.223590:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767997.223596:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775767997.223600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775767997.223609:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076878464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775767997.223623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223647:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7fb80 x1862023076877568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223661:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076877568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223675:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223679:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767997.223683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767997.223698:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7ce00 x1862023076877696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223713:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076877696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223722:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767997.223749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767997.223764:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7ed80 x1862023076877824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076877824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767997.223844:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767997.223860:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223868:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7f800 x1862023076877952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076877952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076877952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767997.223889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767997.223898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7ea00 x1862023076878080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223911:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076878080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223917:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767997.223923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775767997.223932:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223937:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7df80 x1862023076878208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223944:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076878208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223952:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775767997.223957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767997.223966:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.223971:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7d880 x1862023076878336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.223977:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076878336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.223982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.223984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.223986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767997.223989:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775767997.223997:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775767997.224002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767997/real 1775767997] req@ffffa0434aa7e680 x1862023076878464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767997.224008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076878464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767997.224013:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767997.224015:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775767997.224018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076878464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775767997.415123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767997.415125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767998.247127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775767998.247283:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767998.439091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767998.439112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775767999.271101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775767999.271152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767999.463122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775767999.463122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768000.295125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768000.295158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768000.487152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768000.487243:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768001.319111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768001.319115:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768001.511115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768001.511115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768002.343138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768002.343140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768002.343153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768002.343157:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343167:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:2.0:1775768002.343176:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775768002.343182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768002.343183:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768002.343219:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076878592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768002.343241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768002.343247:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343251:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:2.0:1775768002.343256:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343260:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768002.343270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768002.343272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076878592:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768002.343282:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076878720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768002.343292:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343294:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768002.343296:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343300:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:2.0:1775768002.343302:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343306:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768002.343324:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768002.343327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768002.343333:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076878848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768002.343342:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768002.343347:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343350:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:2.0:1775768002.343354:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343358:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343362:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7e300 x1862023076878592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768002.343362:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768002.343371:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076878976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768002.343379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768002.343380:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076878592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768002.343383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768002.343384:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343388:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:1.0:1775768002.343390:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768002.343391:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768002.343394:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768002.343395:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343397:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076878592:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768002.343409:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768002.343418:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076879104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768002.343419:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768002.343428:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343432:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1775768002.343440:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768002.343440:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343444:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00080000:2.0:1775768002.343449:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768002.343451:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7c000 x1862023076878720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768002.343453:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768002.343458:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768002.343463:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076878720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768002.343468:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076879232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768002.343474:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343478:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768002.343478:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768002.343482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768002.343483:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343487:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00100000:1.0:1775768002.343488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768002.343491:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343494:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768002.343499:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768002.343505:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768002.343507:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076879360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768002.343514:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7f100 x1862023076878848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768002.343516:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768002.343520:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768002.343522:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343526:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00100000:1.0:1775768002.343527:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076878848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768002.343529:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343533:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768002.343535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768002.343537:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768002.343539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768002.343547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768002.343552:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076879488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768002.343561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768002.343564:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768002.343565:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768002.343567:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768002.343571:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4504 00000100:00100000:1.0:1775768002.343573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7c380 x1862023076878976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768002.343574:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768002.343578:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768002.343582:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768002.343585:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076878976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768002.343592:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076879616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768002.343593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343597:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076878976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768002.343616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076879104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768002.343637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343647:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7dc00 x1862023076879104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768002.343666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076879104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768002.343675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343681:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076879104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768002.343686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768002.343705:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7d500 x1862023076879232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768002.343748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076879232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768002.343757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768002.343766:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768002.343781:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343789:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7f480 x1862023076879360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768002.343801:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076879360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768002.343806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343810:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768002.343813:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768002.343822:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343827:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7e300 x1862023076879488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768002.343834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076879488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768002.343839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343840:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768002.343845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768002.343853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768002.343858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768002/real 1775768002] req@ffffa0434aa7e680 x1862023076879616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768002.343864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076879616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768002.343873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768002.343875:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768002.343877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076879616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768002.535118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768002.535118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768003.367133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768003.367174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768003.559116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768003.559116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768004.391189:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768004.391251:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768004.583128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768004.583146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768005.415155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768005.415183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768005.607112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768005.607112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768006.439154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768006.439239:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768006.631151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768006.631158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768007.463170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768007.463176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768007.463186:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768007.463192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768007.463197:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463207:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463213:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768007.463244:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076879744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768007.463264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463266:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768007.463267:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463270:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463286:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463289:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463296:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768007.463303:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076879744:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768007.463321:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076879872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768007.463330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463332:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768007.463334:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463336:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463338:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463340:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463343:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768007.463361:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076880000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768007.463372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768007.463377:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463380:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463384:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768007.463390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768007.463409:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768007.463431:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076880128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768007.463448:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768007.463453:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768007.463461:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7d880 x1862023076879744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768007.463465:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463475:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463482:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463488:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768007.463491:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076879744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.463503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.463507:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.463512:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076879744:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768007.463515:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076880256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768007.463529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463531:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768007.463533:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463536:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463540:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463542:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463547:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768007.463555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076879872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768007.463564:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076880384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768007.463581:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768007.463582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768007.463587:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768007.463590:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768007.463595:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7df80 x1862023076879872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768007.463597:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00100000:1.0:1775768007.463613:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076879872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.463619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.463627:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768007.463628:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768007.463633:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076879872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768007.463640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880000:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768007.463650:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768007.463660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463666:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768007.463684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7ea00 x1862023076880000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768007.463695:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076880512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768007.463699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076880000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.463725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768007.463732:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768007.463733:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768007.463741:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775768007.463745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880000:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768007.463746:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463756:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00100000:1.0:1775768007.463760:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880128:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768007.463761:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768007.463771:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463777:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768007.463786:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076880640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768007.463792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768007.463806:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768007.463807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7f800 x1862023076880128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768007.463813:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768007.463821:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768007.463839:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4509 00000100:00080000:2.0:1775768007.463842:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768007.463847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076880128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768007.463848:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768007.463862:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768007.463876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.463881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.463885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768007.463889:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076880768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768007.463897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768007.463933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768007.463949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7ed80 x1862023076880256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768007.463961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076880256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.463967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.463970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.463971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076880256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768007.463994:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768007.464015:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768007.464022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7ce00 x1862023076880384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768007.464058:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076880384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.464064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.464069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.464074:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768007.464081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768007.464103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768007.464122:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7fb80 x1862023076880512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768007.464134:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076880512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.464143:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.464149:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.464153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768007.464161:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768007.464184:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768007.464196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7c700 x1862023076880640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768007.464209:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076880640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.464216:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.464220:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.464223:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768007.464229:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768007.464259:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768007.464274:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768007/real 1775768007] req@ffffa0434aa7d880 x1862023076880768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768007.464288:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076880768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768007.464297:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768007.464303:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768007.464307:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076880768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768007.655088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768007.655089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768008.487120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768008.487175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768008.679104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768008.679109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768009.511107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768009.511136:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768009.703134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768009.703134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768010.535149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768010.535221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768010.727113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768010.727146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768011.559178:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768011.559202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768011.751161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768011.751162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768012.583134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768012.583142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768012.583143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768012.583148:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583158:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583168:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583174:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768012.583183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768012.583209:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076880896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768012.583234:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768012.583240:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583243:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583246:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583250:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583258:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768012.583268:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076881024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768012.583277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768012.583282:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768012.583288:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076880896:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768012.583288:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583311:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583315:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583321:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768012.583331:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076881152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768012.583341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768012.583346:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583349:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583353:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768012.583357:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768012.583360:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583366:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768012.583375:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076881280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768012.583383:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583387:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768012.583389:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583393:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583397:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768012.583399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7e680 x1862023076880896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768012.583403:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583410:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768012.583423:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076881408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768012.583425:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076880896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.583445:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.583452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583460:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076880896:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768012.583461:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583467:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768012.583471:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768012.583477:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768012.583477:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583483:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583498:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768012.583503:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583503:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768012.583518:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076881536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768012.583520:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7e300 x1862023076881024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768012.583540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768012.583544:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583548:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583553:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768012.583558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076881024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768012.583564:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768012.583572:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768012.583572:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768012.583579:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583584:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768012.583586:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076881664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768012.583597:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583600:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768012.583602:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768012.583605:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00100000:1.0:1775768012.583610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768012.583613:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583620:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768012.583639:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768012.583648:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768012.583653:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076881792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768012.583661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7f480 x1862023076881152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768012.583664:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768012.583678:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768012.583679:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768012.583684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076881152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768012.583684:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4514 00000100:00080000:2.0:1775768012.583690:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768012.583697:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768012.583698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768012.583702:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768012.583703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768012.583715:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076881920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768012.583717:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768012.583746:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.583756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7d500 x1862023076881280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.583780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076881280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.583787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.583789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768012.583798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768012.583815:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.583834:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7dc00 x1862023076881408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.583845:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076881408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.583852:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.583855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583858:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076881408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768012.583863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768012.583881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.583893:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7c380 x1862023076881536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.583917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076881536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.583930:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.583933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583936:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768012.583941:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768012.583958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.583967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7f100 x1862023076881664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.583977:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076881664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.583984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.583987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.583989:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768012.584019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768012.584056:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.584066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7c000 x1862023076881792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.584085:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076881792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.584092:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.584094:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.584098:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768012.584104:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768012.584120:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768012.584128:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768012/real 1775768012] req@ffffa0434aa7e300 x1862023076881920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768012.584138:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076881920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768012.584145:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768012.584147:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768012.584149:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076881920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768012.775092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768012.775095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768013.607098:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768013.607124:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768013.799126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768013.799134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768014.631109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768014.631153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768014.823132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768014.823132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768015.655091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768015.655133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768015.847129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768015.847210:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768016.679084:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768016.679150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768016.871114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768016.871150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768017.703090:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768017.703094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768017.703103:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768017.703106:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768017.703110:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703118:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703122:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768017.703156:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076882048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768017.703180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768017.703194:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768017.703197:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703200:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768017.703204:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076882048:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768017.703205:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703216:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768017.703227:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076882176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768017.703235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768017.703238:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768017.703247:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703251:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703254:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768017.703259:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768017.703272:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076882304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768017.703282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768017.703287:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768017.703290:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00100000:1.0:1775768017.703295:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7e680 x1862023076882048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768017.703295:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703302:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703316:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768017.703326:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076882048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768017.703327:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076882432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768017.703350:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768017.703351:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703356:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768017.703358:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768017.703359:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768017.703364:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00100000:1.0:1775768017.703368:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076882048:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768017.703368:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703374:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703377:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768017.703383:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076882560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768017.703388:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703390:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768017.703391:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768017.703394:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768017.703395:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703399:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703403:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703405:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768017.703410:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076882688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768017.703414:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768017.703416:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768017.703421:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768017.703424:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703427:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703431:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768017.703432:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703437:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768017.703448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7d880 x1862023076882176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768017.703455:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076882816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768017.703458:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076882176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768017.703477:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768017.703478:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768017.703481:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768017.703483:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768017.703484:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768017.703488:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768017.703490:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703496:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768017.703498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882304:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768017.703500:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768017.703506:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768017.703516:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076882944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768017.703522:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768017.703529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768017.703531:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7c700 x1862023076882304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768017.703536:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768017.703540:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768017.703545:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076882304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768017.703547:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4519 00000100:00080000:2.0:1775768017.703554:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768017.703561:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768017.703562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703568:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768017.703569:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768017.703570:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768017.703588:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768017.703596:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076883072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768017.703605:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703613:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7fb80 x1862023076882432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076882432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703628:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703629:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768017.703632:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768017.703641:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7ce00 x1862023076882560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076882560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703656:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703660:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076882560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768017.703670:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768017.703679:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703684:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7ed80 x1862023076882688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076882688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703696:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768017.703701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768017.703709:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7f800 x1862023076882816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076882816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703724:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703725:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768017.703729:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768017.703737:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703744:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7ea00 x1862023076882944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076882944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703760:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076882944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768017.703771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768017.703788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768017.703797:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768017/real 1775768017] req@ffffa0434aa7df80 x1862023076883072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768017.703804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076883072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768017.703808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768017.703810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768017.703812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768017.895123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768017.895124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768018.727101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768018.727171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768018.919123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768018.919213:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768019.751102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768019.751141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768019.943144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768019.943144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768020.775114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768020.775169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768020.967093:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768020.967093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768021.799127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768021.799358:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768021.991111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768021.991112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768022.823088:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768022.823097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768022.823106:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768022.823114:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823121:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823127:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768022.823128:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768022.823156:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076883200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768022.823175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768022.823179:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768022.823181:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823183:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823186:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768022.823193:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768022.823199:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076883200:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768022.823204:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076883328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768022.823210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768022.823214:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768022.823231:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823234:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823236:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768022.823240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768022.823246:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768022.823249:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076883456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768022.823256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823259:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768022.823260:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768022.823262:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823264:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823266:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768022.823270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768022.823278:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7df80 x1862023076883200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768022.823278:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076883584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768022.823306:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823308:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768022.823309:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768022.823313:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076883200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823313:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823318:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823323:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768022.823328:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768022.823328:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775768022.823333:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768022.823336:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076883712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768022.823339:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076883200:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768022.823350:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768022.823353:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768022.823356:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823359:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768022.823360:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883328:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768022.823364:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768022.823369:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768022.823378:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076883840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768022.823383:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7ea00 x1862023076883328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768022.823403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768022.823404:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076883328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823408:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823416:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768022.823417:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823423:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768022.823424:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823427:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768022.823433:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883456:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768022.823433:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768022.823451:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076883968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768022.823454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823462:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7f800 x1862023076883456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823465:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823473:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768022.823476:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768022.823478:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076883456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823483:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:1.0:1775768022.823489:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823495:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768022.823495:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823502:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883456:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768022.823507:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768022.823511:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768022.823519:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076884096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768022.823549:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768022.823552:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768022.823555:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768022.823556:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768022.823560:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7ed80 x1862023076883584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768022.823561:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4524 00000100:00080000:2.0:1775768022.823567:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768022.823570:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768022.823573:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768022.823583:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076883584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768022.823587:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076884224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768022.823600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823605:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768022.823613:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768022.823625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823630:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7ce00 x1862023076883712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768022.823641:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076883712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768022.823646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823647:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823649:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076883712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768022.823659:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768022.823668:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823673:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7fb80 x1862023076883840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768022.823680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076883840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768022.823684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823685:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823687:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768022.823690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768022.823697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7c700 x1862023076883968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768022.823708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076883968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768022.823716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076883968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768022.823722:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768022.823729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7d880 x1862023076884096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768022.823739:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076884096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768022.823743:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823744:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823746:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768022.823749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768022.823756:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768022.823760:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768022/real 1775768022] req@ffffa0434aa7e680 x1862023076884224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768022.823766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076884224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768022.823770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768022.823771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768022.823773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768023.015089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768023.015089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768023.847127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768023.847162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768024.039129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768024.039139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768024.871118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768024.871158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768025.063095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768025.063098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768025.895099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768025.895133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768026.087123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768026.087141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768026.919122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768026.919175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768027.111139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768027.111139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768027.943108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768027.943114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768027.943116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768027.943122:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943131:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943141:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943147:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768027.943151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768027.943183:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076884352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768027.943224:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943228:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943232:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943235:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943242:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076884352:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768027.943243:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768027.943256:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076884480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943266:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768027.943267:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943270:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943273:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943275:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768027.943278:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768027.943285:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076884608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943291:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943293:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768027.943295:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943297:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943299:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943302:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768027.943305:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768027.943314:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768027.943327:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076884736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943340:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768027.943343:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768027.943345:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7e300 x1862023076884352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768027.943349:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943357:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943365:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768027.943370:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768027.943376:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076884352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768027.943381:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076884864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943394:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768027.943398:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943403:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943407:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768027.943408:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768027.943412:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768027.943414:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768027.943417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768027.943420:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076884352:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768027.943428:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076884992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943434:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943437:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768027.943443:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768027.943446:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884480:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768027.943448:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943454:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943457:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768027.943461:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768027.943472:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076885120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768027.943495:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943497:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768027.943498:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943501:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943505:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943509:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768027.943511:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768027.943515:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768027.943525:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076885248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768027.943531:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7c000 x1862023076884480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768027.943535:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768027.943543:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768027.943546:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768027.943553:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4529 00000100:00080000:2.0:1775768027.943557:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768027.943561:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076884480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768027.943565:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768027.943574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768027.943582:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076885376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768027.943583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768027.943602:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768027.943619:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943626:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7f100 x1862023076884608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076884608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943645:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943647:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768027.943654:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768027.943665:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943671:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7c380 x1862023076884736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076884736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943688:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768027.943695:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768027.943704:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943709:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7dc00 x1862023076884864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943715:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076884864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943729:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076884864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768027.943733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768027.943743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7d500 x1862023076884992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076884992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076884992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768027.943779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768027.943789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7f480 x1862023076885120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076885120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943817:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768027.943820:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768027.943830:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7e300 x1862023076885248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076885248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943851:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768027.943854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768027.943862:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768027.943868:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768027/real 1775768027] req@ffffa0434aa7e680 x1862023076885376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768027.943875:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076885376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768027.943879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768027.943881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768027.943883:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076885376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768028.135102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768028.135131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768028.967193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768028.967205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768029.159190:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768029.159191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768029.991172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768029.991280:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768030.183101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768030.183157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768031.015124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768031.015155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768031.207119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768031.207128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768032.039138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768032.039310:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768032.231092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768032.231092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768033.063104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775768033.063114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768033.063116:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768033.063118:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768033.063126:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063134:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063140:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768033.063171:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076885504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768033.063196:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063200:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063203:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063206:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768033.063227:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076885632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768033.063228:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076885504:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768033.063242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768033.063246:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063249:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063252:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063254:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768033.063266:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076885760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768033.063276:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063279:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063282:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063284:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768033.063294:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076885888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063300:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063302:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768033.063304:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063307:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063310:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063313:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063316:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768033.063323:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768033.063326:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076886016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063337:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063339:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768033.063341:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063344:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063347:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063349:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063353:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768033.063354:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768033.063366:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076886144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063372:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063375:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768033.063377:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885888:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768033.063378:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063385:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063389:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063391:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063395:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768033.063406:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768033.063407:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076886272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063418:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063420:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768033.063421:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063441:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00100000:1.0:1775768033.063444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768033.063447:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768033.063458:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063467:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768033.063470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768033.063485:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076886400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768033.063499:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768033.063502:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768033.063505:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768033.063510:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4534 00000100:00080000:2.0:1775768033.063515:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768033.063516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886400:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768033.063524:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768033.063530:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768033.063546:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076886528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768033.063582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768033.255088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768033.255104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768034.087126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768034.087250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768034.279094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768034.279114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768035.111150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768035.111220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768035.303115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768035.303115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768036.135112:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768036.135355:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775768036.135459:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775768036.135500:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775768036.135517:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135560:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135574:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135585:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135592:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135600:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135610:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135620:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768036.135630:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:2.0:1775768036.135646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7e680 x1862023076886528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135682:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076886528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135711:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768036.135721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7df80 x1862023076886400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135736:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076886400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768036.135765:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7ea00 x1862023076886272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135783:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076886272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768036.135808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7ed80 x1862023076886016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076886016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135838:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768036.135846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7f800 x1862023076886144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076886144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135868:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135875:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076886144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768036.135883:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7c700 x1862023076885632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076885632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135905:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768036.135927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7fb80 x1862023076885760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076885760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768036.135963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7ce00 x1862023076885888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.135976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076885888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.135984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.135987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.135990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076885888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768036.135998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768033/real 1775768036] req@ffffa0434aa7d880 x1862023076885504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768036.136010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076885504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768036.136018:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768036.136021:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768036.136025:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076885504:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768036.327119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768036.327121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768037.159133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768037.159194:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768037.351135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768037.351135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768038.183127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768038.183157:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183163:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768038.183167:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183187:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183195:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768038.183239:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076886656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775768038.183247:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768038.183278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183282:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768038.183285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183289:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183294:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183298:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183308:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768038.183322:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076886784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183333:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183337:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768038.183339:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183343:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183346:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183350:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183361:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768038.183371:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076886912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768038.183421:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183425:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183429:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183433:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183439:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768038.183450:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076887040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183460:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768038.183465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183468:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183472:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183476:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183480:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768038.183489:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076887168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768038.183504:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183508:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183511:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183515:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183520:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768038.183529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076887296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183539:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768038.183549:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183554:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183557:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183561:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768038.183576:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076887424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183585:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183589:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768038.183591:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183594:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183598:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183602:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183606:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768038.183615:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076887552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768038.183631:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768038.183634:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768038.183636:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768038.183640:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4539 00000100:00080000:0.0:1775768038.183644:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768038.183648:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768038.183653:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768038.183663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076887680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768038.183750:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076886656:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775768038.183826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.183867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349641500 x1862023076886656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.183888:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076886656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.183899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.183904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.183909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076886656:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768038.183918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768038.183940:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.183952:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349641880 x1862023076886784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.183965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076886784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.183974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.183978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.183982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768038.183987:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768038.184005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349640e00 x1862023076886912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184050:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076886912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184067:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076886912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768038.184073:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768038.184092:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184109:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349642680 x1862023076887040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076887040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184138:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768038.184144:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768038.184161:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349641c00 x1862023076887168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076887168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184192:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184196:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768038.184205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768038.184223:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184234:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349640700 x1862023076887296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184254:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076887296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184267:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184270:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768038.184276:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768038.184295:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184305:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349641180 x1862023076887424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184318:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076887424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184327:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184330:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184333:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768038.184340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768038.184357:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184367:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349640a80 x1862023076887552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184402:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076887552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184415:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184418:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184421:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768038.184426:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768038.184447:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768038.184458:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768038/real 1775768038] req@ffffa04349643800 x1862023076887680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768038.184469:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076887680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768038.184477:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768038.184479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768038.184482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076887680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768038.375143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768038.375143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768039.207269:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768039.207298:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768039.399129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768039.399140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768040.231109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768040.231188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768040.423108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768040.423108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768041.255099:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768041.255145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768041.447134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768041.447142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768042.279119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768042.279190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768042.471128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768042.471211:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768043.303203:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1775768043.303211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768043.303211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768043.303215:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768043.303225:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303237:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303249:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303257:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768043.303302:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076887808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303349:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768043.303351:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303355:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303359:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303362:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303379:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768043.303394:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076887936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303405:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768043.303407:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303409:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303412:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768043.303419:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076887808:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768043.303420:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768043.303442:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076888064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303454:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768043.303456:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303459:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303462:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303465:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303470:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768043.303480:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076888192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303495:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768043.303498:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303505:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303512:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768043.303513:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768043.303522:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303530:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768043.303548:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076888320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303556:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303560:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768043.303563:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303573:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303578:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768043.303579:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7d880 x1862023076887808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768043.303592:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303603:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768043.303620:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076888448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303630:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303632:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768043.303635:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303639:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303643:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303646:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768043.303666:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076888576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303677:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768043.303685:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076887808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768043.303686:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303706:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303711:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303716:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303722:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768043.303735:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076888704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768043.303746:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768043.303749:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768043.303751:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768043.303755:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4544 00000100:00080000:2.0:1775768043.303759:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768043.303763:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768043.303769:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768043.303782:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076888832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768043.303822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.303833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.303839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076887808:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768043.303923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768043.303947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.303958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7ce00 x1862023076887936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.303968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076887936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.303975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.303978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.303980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076887936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768043.303989:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768043.304009:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304022:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7fb80 x1862023076888064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304067:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076888064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304076:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304078:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304081:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768043.304088:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768043.304115:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304127:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7c700 x1862023076888192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076888192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304145:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304147:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768043.304151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768043.304163:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304169:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7f800 x1862023076888320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304177:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076888320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304186:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076888320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768043.304190:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768043.304203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304208:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7ed80 x1862023076888448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304217:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076888448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304222:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304224:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304226:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768043.304229:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768043.304239:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304244:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7ea00 x1862023076888576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304251:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076888576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304261:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304263:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304265:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768043.304269:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768043.304278:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304284:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7df80 x1862023076888704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304290:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076888704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304295:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304297:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768043.304303:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768043.304316:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768043.304337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768043/real 1775768043] req@ffffa0434aa7e680 x1862023076888832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768043.304349:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076888832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768043.304358:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768043.304362:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768043.304366:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076888832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768043.495158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768043.495158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768044.327130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768044.327259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768044.519108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768044.519109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768045.351137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768045.351193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768045.543133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768045.543133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768046.375110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768046.375179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768046.567090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768046.567091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768047.399148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768047.399320:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768047.591130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768047.591130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768048.423163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768048.423168:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423179:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768048.423183:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423203:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423218:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768048.423266:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076888960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775768048.423292:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768048.423312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768048.423318:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423324:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423329:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423333:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423344:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768048.423348:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076888960:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768048.423361:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076889088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423374:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423376:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768048.423378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423382:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423386:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423391:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423397:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768048.423408:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076889216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768048.423426:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768048.423430:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768048.423433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423437:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423442:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423445:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768048.423463:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076889344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423470:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423472:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768048.423474:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423477:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423480:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768048.423486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7e300 x1862023076888960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768048.423486:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423494:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768048.423507:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076889472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423518:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423523:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1775768048.423524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076888960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768048.423530:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423543:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:0.0:1775768048.423553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768048.423553:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768048.423564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768048.423565:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423577:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768048.423578:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076888960:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768048.423594:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076889600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423603:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423605:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768048.423607:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423609:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423612:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423614:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423618:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768048.423624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768048.423628:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076889728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423642:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768048.423645:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768048.423646:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423651:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423656:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423658:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423661:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775768048.423669:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768048.423678:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076889856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768048.423686:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768048.423688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7f480 x1862023076889088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768048.423689:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768048.423692:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768048.423698:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4550 00000100:00080000:2.0:1775768048.423705:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768048.423714:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768048.423720:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768048.423732:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076889984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768048.423738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076889088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.423763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.423768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.423773:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768048.423783:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768048.423807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.423815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7d500 x1862023076889216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.423822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076889216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.423827:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.423830:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.423832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768048.423836:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768048.423847:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.423853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7dc00 x1862023076889344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.423860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076889344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.423865:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.423867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.423869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768048.423872:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768048.423881:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.423886:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7c380 x1862023076889472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.423891:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076889472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.423896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.423898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.423900:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076889472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768048.423904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768048.423913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.423917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7f100 x1862023076889600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.423925:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076889600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.423934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.423937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.423940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768048.423962:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768048.423980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.423988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7c000 x1862023076889728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.423995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076889728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.424000:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.424002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.424004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768048.424007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768048.424016:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.424021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7e300 x1862023076889856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.424049:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076889856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.424054:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.424056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.424059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768048.424063:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768048.424073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768048.424079:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768048/real 1775768048] req@ffffa0434aa7e680 x1862023076889984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768048.424086:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076889984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768048.424090:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768048.424092:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768048.424093:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076889984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768048.615148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768048.615149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768049.447184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768049.447262:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768049.639141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768049.639141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768050.471123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768050.471211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768050.663166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768050.663166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768051.495137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768051.495203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768051.687149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768051.687149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768052.519114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768052.519140:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768052.711141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768052.711141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768053.543150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768053.543152:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543163:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768053.543168:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543184:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543193:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543200:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768053.543248:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076890112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543282:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768053.543284:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543292:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543296:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543307:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00010000:00010000:2.0:1775768053.543311:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768053.543321:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076890240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543333:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768053.543335:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543338:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543342:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543344:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543349:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768053.543359:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076890368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543367:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768053.543368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076890112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768053.543370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768053.543371:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543374:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543377:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543380:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768053.543390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076890496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543397:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543399:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768053.543401:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543403:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543408:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543411:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543414:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775768053.543416:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768053.543422:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076890624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543432:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768053.543433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543436:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543439:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543442:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543446:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768053.543453:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076890752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768053.543456:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350932680 x1862023076890112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768053.543461:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768053.543466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543468:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543471:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768053.543473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076890112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768053.543474:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768053.543482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768053.543485:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076890880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768053.543486:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076890112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768053.543492:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768053.543495:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775768053.543496:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890240:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768053.543496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543499:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00080000:1.0:1775768053.543502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543505:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543508:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775768053.543514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768053.543516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076891008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768053.543523:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768053.543524:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350930000 x1862023076890240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768053.543527:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768053.543528:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768053.543531:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4555 00000100:00100000:0.0:1775768053.543534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076890240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768053.543534:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768053.543537:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768053.543541:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775768053.543542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543548:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768053.543548:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076891136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768053.543570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768053.543588:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543596:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350930a80 x1862023076890368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543603:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076890368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543608:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543609:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543611:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768053.543615:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768053.543624:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543629:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350931c00 x1862023076890496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076890496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543655:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543659:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768053.543662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768053.543673:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350932300 x1862023076890624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543692:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076890624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543699:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076890624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768053.543710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076890752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768053.543726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543734:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350930e00 x1862023076890752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543749:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076890752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543761:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076890752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768053.543764:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076890880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768053.543775:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350931880 x1862023076890880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543787:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076890880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543797:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076890880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768053.543804:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768053.543820:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350932a00 x1862023076891008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543839:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076891008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543854:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768053.543869:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768053.543887:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768053.543896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768053/real 1775768053] req@ffffa04350930700 x1862023076891136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768053.543914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076891136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768053.543919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768053.543921:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768053.543922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768053.735121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768053.735121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768054.567183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768054.567253:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768054.759186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768054.759203:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768055.591116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768055.591203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768055.783124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768055.783124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768056.615092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768056.615107:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768056.807130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768056.807130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768057.639168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768057.639208:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768057.831171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768057.831171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768058.663161:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663171:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775768058.663172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768058.663189:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663202:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663213:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663220:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768058.663238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768058.663269:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076891264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663292:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663295:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768058.663296:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663299:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663302:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663306:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768058.663324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076891264:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768058.663325:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076891392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663335:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663337:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768058.663338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663340:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663342:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663345:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768058.663355:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076891520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663360:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663363:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768058.663364:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663367:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663381:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663384:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775768058.663393:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768058.663395:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076891648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663402:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663404:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768058.663406:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663408:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663410:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663413:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768058.663421:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076891776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663426:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663429:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768058.663430:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663433:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663436:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768058.663440:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350930700 x1862023076891264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768058.663442:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663448:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768058.663458:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076891904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768058.663462:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076891264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768058.663475:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663477:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768058.663479:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768058.663479:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768058.663487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768058.663487:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663490:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768058.663493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076891264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768058.663496:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663501:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768058.663508:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076892032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663514:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663515:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768058.663516:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768058.663518:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663521:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768058.663524:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891392:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768058.663525:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663532:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768058.663540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076892160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768058.663546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768058.663549:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768058.663551:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768058.663553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768058.663556:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4560 00000100:00080000:1.0:1775768058.663559:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768058.663562:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768058.663564:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768058.663571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350932a00 x1862023076891392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768058.663574:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076892288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768058.663610:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076891392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663623:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663626:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768058.663632:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768058.663651:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663660:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350931880 x1862023076891520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663670:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076891520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663677:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768058.663689:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768058.663708:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350930e00 x1862023076891648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663729:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076891648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663737:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663744:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768058.663749:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768058.663775:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663786:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350932300 x1862023076891776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663798:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076891776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663806:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076891776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768058.663817:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768058.663831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663838:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350931c00 x1862023076891904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663844:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076891904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663854:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663856:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076891904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768058.663860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768058.663869:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350930a80 x1862023076892032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663880:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076892032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663884:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768058.663902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768058.663911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663915:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350930000 x1862023076892160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663921:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076892160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663929:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768058.663932:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768058.663941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768058.663945:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768058/real 1775768058] req@ffffa04350932680 x1862023076892288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768058.663951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076892288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768058.663956:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768058.663957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768058.663959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076892288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768058.855109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768058.855120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768059.687081:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768059.687191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768059.879128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768059.879128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768060.711115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768060.711199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768060.903167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768060.903179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768061.735158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768061.735214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768061.927111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768061.927126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768062.759167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768062.759226:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768062.951131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768062.951145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768063.783341:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768063.783381:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768063.785673:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.785681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768063.785684:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.785692:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.785700:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.785706:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768063.785735:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076892416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.785755:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.785758:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768063.785760:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.785762:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.785770:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.785775:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.785783:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768063.785790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076892416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768063.785795:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076892544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.785803:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.785806:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768063.785807:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.785810:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.785812:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.785815:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.785818:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768063.785828:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076892672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.785836:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.785839:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768063.785841:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.785846:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4565 00000800:00000100:0.0:1775768063.785851:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768063.785851:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.785858:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.785864:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768063.785879:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076892800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.785886:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.785888:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768063.785889:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.785894:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00100000:0.0:1775768063.785899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7df80 x1862023076892416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768063.785900:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.785907:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.785912:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768063.785928:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076892928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768063.785929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076892416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.785959:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768063.785960:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768063.785967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.785970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076892416:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768063.785973:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768063.785978:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768063.785988:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892544:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768063.785989:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.785997:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.786003:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.786007:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768063.786017:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076893056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768063.786024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768063.786053:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.786057:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768063.786058:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.786062:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00100000:0.0:1775768063.786065:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7ea00 x1862023076892544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768063.786067:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.786074:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.786081:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768063.786091:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076892544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768063.786092:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076893184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.786111:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.786112:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768063.786113:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768063.786119:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.786123:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768063.786124:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768063.786139:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768063.786141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892544:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768063.786147:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768063.786156:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768063.786165:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076893312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768063.786174:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768063.786176:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768063.786177:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768063.786182:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768063.786182:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4565 00000100:00080000:2.0:1775768063.786188:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768063.786191:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768063.786194:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768063.786197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7ed80 x1862023076892672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768063.786204:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076893440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768063.786208:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076892672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786220:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786223:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786228:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768063.786235:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768063.786254:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786264:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7f800 x1862023076892800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786277:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076892800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786282:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786284:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786285:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768063.786289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768063.786301:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786307:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7c700 x1862023076892928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786313:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076892928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786318:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786320:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786322:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076892928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768063.786336:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768063.786345:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7fb80 x1862023076893056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786357:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076893056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786365:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786367:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786369:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768063.786371:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768063.786379:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786383:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7ce00 x1862023076893184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786389:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076893184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786394:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786395:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786397:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768063.786399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768063.786407:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786411:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7d880 x1862023076893312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076893312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786422:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786423:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768063.786428:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768063.786436:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768063.786440:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768063/real 1775768063] req@ffffa0434aa7df80 x1862023076893440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768063.786446:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076893440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768063.786450:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768063.786452:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768063.786453:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076893440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768063.975084:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768063.975151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768064.807160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768064.807371:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768064.999143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768064.999143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768065.831154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768065.831211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768066.023175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768066.023176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768066.855191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768066.855259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768067.047182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768067.047182:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768067.879180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768067.879195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768068.071167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768068.071184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768068.903190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768068.903250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768068.903267:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1775768068.903274:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768068.903277:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903287:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903293:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768068.903329:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076893568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768068.903363:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903366:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903368:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903371:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903379:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768068.903388:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076893696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903397:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903400:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775768068.903405:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076893568:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768068.903406:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903424:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903429:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903435:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903444:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768068.903460:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076893824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903470:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903472:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768068.903473:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903476:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903488:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903491:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903496:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775768068.903498:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768068.903511:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076893952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903520:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903522:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768068.903524:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903529:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903531:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903533:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903537:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768068.903551:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076894080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768068.903559:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350932680 x1862023076893568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768068.903561:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768068.903583:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903594:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903599:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768068.903606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076893568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768068.903606:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903618:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768068.903635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.903643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768068.903644:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076894208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768068.903652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076893568:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768068.903691:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903693:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768068.903694:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903699:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903702:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768068.903704:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903711:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768068.903713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768068.903730:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076894336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903739:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768068.903746:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768068.903755:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768068.903759:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768068.903765:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903768:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768068.903771:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350930000 x1862023076893696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768068.903777:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903786:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768068.903796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076893696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768068.903802:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076894464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768068.903841:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768068.903844:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768068.903846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768068.903846:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768068.903860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768068.903861:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4570 00000100:00080000:1.0:1775768068.903869:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768068.903872:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893696:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768068.903880:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768068.903884:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768068.903887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768068.903898:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076894592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768068.903911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.903921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350930a80 x1862023076893824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.903930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076893824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.903937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.903940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.903942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768068.903947:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768068.903958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.903963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350931c00 x1862023076893952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.903981:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076893952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.903990:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.903993:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.903997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076893952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768068.904003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768068.904057:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.904069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350932300 x1862023076894080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.904092:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076894080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.904099:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.904101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.904104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768068.904108:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768068.904128:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.904139:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350930e00 x1862023076894208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.904151:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076894208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.904167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.904171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.904175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768068.904202:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768068.904223:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.904235:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350931880 x1862023076894336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.904248:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076894336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.904257:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.904261:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.904264:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768068.904269:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768068.904289:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.904301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350932a00 x1862023076894464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.904313:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076894464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.904322:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.904326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.904330:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768068.904337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768068.904357:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768068.904369:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768068/real 1775768068] req@ffffa04350930700 x1862023076894592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768068.904382:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076894592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768068.904391:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768068.904394:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768068.904397:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076894592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768069.095137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768069.095142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768069.927150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768069.927211:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768070.119173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768070.119173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768070.951148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768070.951235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768071.143149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768071.143182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768071.975250:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768071.975255:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768072.167173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768072.167185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768072.999142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768072.999175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768073.191137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768073.191137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768074.023143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768074.023159:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768074.023166:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768074.023171:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768074.023182:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023194:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775768074.023196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768074.023202:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768074.023244:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076894720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768074.023283:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768074.023287:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768074.023289:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768074.023294:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023299:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768074.023304:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768074.023313:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076894720:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768074.023315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768074.023335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076894848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768074.023347:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768074.023351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768074.023362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768074.023368:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4575 00000800:00000100:0.0:1775768074.023373:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768074.023375:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768074.023383:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768074.023389:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768074.023406:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076894976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768074.023417:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768074.023419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7e680 x1862023076894720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023424:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768074.023428:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768074.023436:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023442:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768074.023443:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076894720/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023451:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768074.023459:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023466:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768074.023470:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768074.023481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076894720:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768074.023486:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076895104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768074.023499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768074.023503:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768074.023505:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768074.023510:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00100000:0.0:1775768074.023513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768074.023516:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768074.023523:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768074.023529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1775768074.023538:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768074.023549:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7e300 x1862023076894848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768074.023556:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076895232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768074.023563:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076894848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768074.023582:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768074.023583:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768074.023586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768074.023592:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894848:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768074.023593:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768074.023598:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768074.023601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768074.023607:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023612:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768074.023618:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768074.023620:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768074.023627:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768074.023629:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7c000 x1862023076894976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768074.023639:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076894976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768074.023645:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076895360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768074.023657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768074.023667:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768074.023674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076894976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768074.023675:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768074.023679:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768074.023685:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076895104:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768074.023687:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023695:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768074.023700:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768074.023706:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768074.023721:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076895488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768074.023726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768074.023738:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768074.023739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7f100 x1862023076895104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023748:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768074.023753:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768074.023756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076895104/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023762:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023771:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768074.023772:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023778:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768074.023781:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768074.023784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076895104:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775768074.023794:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768074.023802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076895232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768074.023812:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076895616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768074.023826:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768074.023829:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768074.023834:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768074.023837:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768074.023841:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7c380 x1862023076895232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023846:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4575 00000100:00080000:2.0:1775768074.023856:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768074.023858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076895232/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768074.023866:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768074.023870:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768074.023875:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768074.023878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076895232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768074.023884:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768074.023894:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076895744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768074.023900:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768074.023909:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7dc00 x1862023076895360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768074.023936:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076895360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768074.023943:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023946:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768074.023948:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768074.023952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768074.023969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768074.023976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7d500 x1862023076895488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768074.023983:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076895488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768074.023988:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.023997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768074.024000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768074.024012:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768074.024024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768074.024072:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7f480 x1862023076895616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768074.024085:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076895616/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768074.024101:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.024104:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768074.024107:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768074.024113:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768074.024129:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768074.024137:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768074/real 1775768074] req@ffffa0434aa7f100 x1862023076895744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768074.024144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076895744/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768074.024149:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768074.024151:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768074.024153:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076895744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768074.215120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768074.215120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768075.047149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768075.047215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768075.239146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768075.239152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768076.071177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768076.071263:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768076.263168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768076.263168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768077.095225:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768077.095275:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768077.287161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768077.287161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768078.119170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768078.119293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768078.311187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768078.311187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768079.143208:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768079.143214:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143223:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775768079.143227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768079.143228:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143241:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143252:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143261:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768079.143305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076895872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143342:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768079.143344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143349:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143354:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143359:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768079.143367:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076895872:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775768079.143370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768079.143387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076896000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143400:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143404:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768079.143407:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143412:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143417:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143421:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768079.143439:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076896128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143450:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143454:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768079.143456:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768079.143457:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768079.143461:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143464:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143468:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143473:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768079.143484:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076896256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143494:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143497:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768079.143500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768079.143500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349643b80 x1862023076895872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768079.143504:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143508:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143512:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768079.143522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076895872/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768079.143528:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076896384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768079.143534:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768079.143537:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143541:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768079.143542:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143547:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143550:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143554:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143558:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768079.143568:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076896512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143578:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143581:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768079.143583:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143586:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143590:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143593:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143598:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768079.143608:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076896640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768079.143608:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.143613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076895872:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768079.143617:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143620:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768079.143622:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143625:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4580 00000100:00080000:0.0:1775768079.143628:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768079.143631:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143636:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768079.143639:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768079.143644:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076896768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768079.143653:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768079.143657:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768079.143659:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768079.143662:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4580 00000800:00000100:3.0:1775768079.143662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768079.143674:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768079.143675:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349640380 x1862023076896000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768079.143678:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768079.143683:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768079.143694:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076896896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768079.143696:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076896000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.143707:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.143710:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.143713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768079.143720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768079.143742:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.143751:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349643100 x1862023076896128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.143760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076896128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.143906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.143911:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.143916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768079.143923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768079.143948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.143959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349642300 x1862023076896256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144151:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076896256/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144163:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144165:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144167:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768079.144172:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768079.144189:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.144197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349642a00 x1862023076896384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144205:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076896384/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144211:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144213:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144215:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076896384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768079.144218:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768079.144229:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.144234:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349643800 x1862023076896512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144241:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076896512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144246:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144248:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144251:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768079.144254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768079.144265:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.144269:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349640a80 x1862023076896640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144276:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076896640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144281:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144283:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144285:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768079.144302:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768079.144311:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.144317:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349641180 x1862023076896768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144324:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076896768/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144329:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144331:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144333:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768079.144337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768079.144346:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768079.144352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768079/real 1775768079] req@ffffa04349640700 x1862023076896896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768079.144359:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076896896/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768079.144364:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768079.144366:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768079.144367:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076896896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768079.335114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768079.335114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768080.167160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768080.167321:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768080.359125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768080.359128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768081.191088:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768081.191175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768081.383085:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768081.383099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768082.215121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768082.215191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768082.407128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768082.407128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768083.239134:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768083.239135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768083.431098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768083.431102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768084.263133:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768084.263142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768084.263143:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768084.263149:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768084.263157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768084.263160:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263172:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263179:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768084.263213:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076897024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263235:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263238:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768084.263239:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263242:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263245:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263248:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263256:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768084.263267:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076897024:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768084.263269:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076897152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263283:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768084.263284:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263290:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263292:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263296:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768084.263305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076897280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263313:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768084.263315:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263317:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263319:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263321:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768084.263326:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768084.263335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076897408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263342:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263344:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768084.263346:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263348:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263351:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263353:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768084.263364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349641c00 x1862023076897024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768084.263367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076897536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263391:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768084.263395:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263398:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263401:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768084.263404:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076897024/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768084.263406:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768084.263422:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076897664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768084.263426:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768084.263430:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263432:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768084.263435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768084.263436:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768084.263441:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00100000:3.0:1775768084.263444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076897024:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768084.263445:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263452:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263459:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768084.263466:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076897792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263474:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768084.263475:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263481:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00100000:3.0:1775768084.263481:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768084.263486:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768084.263489:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263492:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768084.263501:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076897920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768084.263507:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768084.263510:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768084.263511:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768084.263513:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4585 00000100:00080000:0.0:1775768084.263516:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768084.263520:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768084.263522:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768084.263527:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768084.263534:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076898048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768084.263542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349642680 x1862023076897152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263559:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076897152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263572:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768084.263577:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768084.263593:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263600:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349640e00 x1862023076897280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076897280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263625:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263627:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768084.263631:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768084.263644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349641880 x1862023076897408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076897408/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263676:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263679:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768084.263690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768084.263710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349641500 x1862023076897536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076897536/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263743:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263747:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263750:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076897536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768084.263757:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768084.263776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349640700 x1862023076897664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076897664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768084.263824:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768084.263844:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349641180 x1862023076897792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263868:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076897792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263887:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768084.263918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768084.263956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.263966:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349640a80 x1862023076897920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.263977:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076897920/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.263984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.263987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.263990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076897920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768084.263996:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768084.264021:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768084.264052:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768084/real 1775768084] req@ffffa04349643800 x1862023076898048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768084.264061:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076898048/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768084.264066:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768084.264068:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768084.264070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768084.455128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768084.455133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768085.287163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768085.287171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768085.479117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768085.479117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768086.311125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768086.311181:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768086.503158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768086.503160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768087.335108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768087.335150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768087.527148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768087.527148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768088.359123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768088.359156:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768088.551127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768088.551127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768089.383104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768089.383108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768089.383113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768089.383117:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383127:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4591 00010000:00010000:3.0:1775768089.383133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768089.383135:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768089.383143:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768089.383181:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076898176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768089.383211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768089.383217:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383223:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00080000:0.0:1775768089.383228:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768089.383232:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383238:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076898176:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775768089.383246:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768089.383265:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076898304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768089.383277:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383281:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768089.383284:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383288:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00080000:0.0:1775768089.383294:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768089.383294:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383302:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768089.383308:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768089.383323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076898432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768089.383331:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349642a00 x1862023076898176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768089.383353:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383360:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00080000:0.0:1775768089.383365:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768089.383365:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076898176/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383375:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768089.383377:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383384:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775768089.383385:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768089.383388:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076898176:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768089.383408:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768089.383413:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076898560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768089.383426:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768089.383429:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383434:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768089.383437:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383444:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00100000:3.0:1775768089.383445:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349642300 x1862023076898304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383452:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768089.383459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076898304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383461:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768089.383471:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775768089.383472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383478:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383481:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768089.383486:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768089.383490:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076898688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768089.383502:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768089.383505:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383510:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768089.383513:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768089.383515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349643100 x1862023076898432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383523:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00080000:0.0:1775768089.383531:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768089.383532:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076898432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383541:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768089.383552:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768089.383553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383558:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383561:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768089.383570:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076898816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768089.383576:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898560:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768089.383592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:3.0:1775768089.383598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383600:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383607:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00100000:3.0:1775768089.383609:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349640380 x1862023076898560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383615:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768089.383623:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383624:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076898560/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768089.383633:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768089.383634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383640:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768089.383647:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768089.383653:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076898944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768089.383663:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383668:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383672:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768089.383674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349643b80 x1862023076898688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383675:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383685:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00100000:3.0:1775768089.383689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076898688/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383692:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768089.383701:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768089.383701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775768089.383711:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768089.383721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076898688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768089.383728:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076899072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768089.383733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768089.383747:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768089.383750:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768089.383754:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768089.383757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349642a00 x1862023076898816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383758:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768089.383769:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4591 00000100:00100000:3.0:1775768089.383774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076898816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768089.383775:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768089.383783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383785:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383790:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768089.383794:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898944:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768089.383804:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768089.383811:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349643800 x1862023076898944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768089.383824:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076899200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768089.383837:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076898944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768089.383847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383850:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383852:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076898944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768089.383863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768089.383874:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349640a80 x1862023076899072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768089.383911:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076899072/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768089.383916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768089.383923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768089.383937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768089.383943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768089/real 1775768089] req@ffffa04349641180 x1862023076899200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768089.383950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076899200/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768089.383955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768089.383957:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768089.383959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768089.575136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768089.575136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768090.407121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768090.407165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768090.599111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768090.599122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768091.431139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768091.431221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768091.623170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768091.623210:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768092.455141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768092.455204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768092.647093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768092.647120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768093.479113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768093.479148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768093.671137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768093.671137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768094.503120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768094.503120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768094.503133:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503141:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503152:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503161:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768094.503162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768094.503200:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076899328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768094.503228:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503231:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503234:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503238:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503246:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768094.503258:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076899456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768094.503262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076899328:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768094.503270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768094.503274:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503277:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503279:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503281:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768094.503292:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076899584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503298:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503301:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768094.503302:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503303:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503314:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503317:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503321:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768094.503329:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768094.503330:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076899712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503341:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768094.503342:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503345:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503348:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503351:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503354:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768094.503361:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076899840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503373:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768094.503375:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7c000 x1862023076899328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768094.503376:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768094.503381:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503388:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503393:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503399:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503402:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768094.503410:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076899328/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768094.503411:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076899968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503444:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503446:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768094.503446:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768094.503450:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768094.503455:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768094.503456:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503460:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503465:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503465:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076899328:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768094.503468:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768094.503478:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076900096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768094.503483:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503486:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768094.503487:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503493:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00100000:3.0:1775768094.503494:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899456:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768094.503495:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768094.503506:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503509:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768094.503519:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076900224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768094.503527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768094.503529:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768094.503532:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768094.503534:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768094.503538:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4596 00000100:00080000:2.0:1775768094.503542:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768094.503546:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7e300 x1862023076899456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768094.503547:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768094.503556:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768094.503570:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076899456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768094.503574:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076900352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768094.503592:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503599:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503605:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768094.503613:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768094.503632:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503642:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7e680 x1862023076899584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503654:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076899584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503663:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503666:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768094.503675:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768094.503693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503704:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7df80 x1862023076899712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503716:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076899712/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503724:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503730:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768094.503736:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768094.503753:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7d880 x1862023076899840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076899840/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503796:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076899840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768094.503802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768094.503818:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7ce00 x1862023076899968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503855:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076899968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503864:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076899968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768094.503876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900096:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768094.503894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7fb80 x1862023076900096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076900096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.503930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768094.503950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768094.503967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.503976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7c700 x1862023076900224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.503987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076900224/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.503996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.503999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.504002:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768094.504008:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768094.504024:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768094.504054:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768094/real 1775768094] req@ffffa0434aa7f800 x1862023076900352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768094.504067:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076900352/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768094.504076:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768094.504079:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768094.504082:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076900352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768094.695107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768094.695108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768095.527132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768095.527160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768095.719140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768095.719150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768096.551101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768096.551135:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768096.743162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768096.743163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768097.575132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768097.575158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768097.767135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768097.767139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768098.599130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768098.599157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768098.791101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768098.791101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768099.623139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768099.623139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768099.623149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768099.623153:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623163:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623173:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775768099.623174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768099.623181:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768099.623217:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076900480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768099.623238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768099.623242:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623245:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623248:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623251:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768099.623268:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076900608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768099.623275:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076900480:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768099.623284:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623287:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768099.623289:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623292:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623296:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623298:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623301:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768099.623311:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076900736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768099.623322:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623325:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768099.623326:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623328:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623331:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623333:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623336:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768099.623344:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076900864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768099.623349:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623352:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768099.623353:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623356:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623358:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623360:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623364:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768099.623370:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076900992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768099.623378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768099.623384:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623389:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00100000:3.0:1775768099.623393:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900608:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768099.623395:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623404:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623411:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768099.623427:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076901120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768099.623435:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768099.623441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768099.623448:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623454:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00100000:3.0:1775768099.623456:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900864:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768099.623460:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623467:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623474:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768099.623478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768099.623493:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076901248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768099.623502:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768099.623507:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623512:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768099.623514:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623520:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00100000:3.0:1775768099.623524:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768099.623525:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623534:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623541:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768099.623554:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076901376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768099.623575:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768099.623578:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768099.623580:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768099.623584:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4601 00000100:00080000:0.0:1775768099.623587:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768099.623591:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768099.623595:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768099.623600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768099.623606:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076901504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768099.623641:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768099.815136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768099.815136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768100.647182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768100.647185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768100.839117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768100.839117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768101.671139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768101.671197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768101.863118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768101.863118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768102.695166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768102.695263:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775768102.695270:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775768102.695302:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768102.695315:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695362:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695373:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695379:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695385:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695392:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695398:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695405:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768102.695411:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775768102.695535:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349643800 x1862023076901504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076901504/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695578:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695583:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768102.695596:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349640a80 x1862023076901376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076901376/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695619:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768102.695628:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349641180 x1862023076901248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695646:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076901248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695653:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695656:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695659:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768102.695666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349641c00 x1862023076901120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076901120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076901120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768102.695696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349641500 x1862023076900608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076900608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768102.695724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349641880 x1862023076900736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076900736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695740:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695743:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695746:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768102.695764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349640e00 x1862023076900864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076900864/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695781:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768102.695791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349642680 x1862023076900992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076900992/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076900992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768102.695817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768099/real 1775768102] req@ffffa04349640700 x1862023076900480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768102.695828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076900480/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768102.695834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768102.695837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768102.695840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076900480:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768102.887098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768102.887102:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768103.719148:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768103.719156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768103.911101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768103.911101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768104.743173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768104.743280:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768104.743309:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743316:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768104.743321:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743331:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743341:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743348:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768104.743384:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076901632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743420:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768104.743422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743426:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743430:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743434:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743443:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768104.743455:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076901760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743465:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743470:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768104.743472:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743476:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743479:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743482:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768104.743494:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076901888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743503:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743506:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768104.743508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743511:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743517:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743521:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768104.743529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076902016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743537:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768104.743541:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743544:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743547:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743550:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743554:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768104.743562:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076902144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743569:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743572:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768104.743574:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743578:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743581:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743584:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743588:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768104.743596:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076902272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743604:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743607:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768104.743608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743611:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743614:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743617:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743621:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768104.743629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076902400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743637:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743639:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768104.743642:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743646:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743654:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743658:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743662:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768104.743670:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076902528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768104.743678:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768104.743681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768104.743683:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768104.743686:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4606 00000100:00080000:2.0:1775768104.743688:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768104.743692:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768104.743696:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768104.743704:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076902656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768104.743807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076901632:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768104.743967:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744006:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7ed80 x1862023076901632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076901632/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744056:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744065:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076901632:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768104.744072:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076901760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768104.744098:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744111:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7ea00 x1862023076901760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076901760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744137:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076901760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768104.744142:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076901888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768104.744162:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744174:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7f800 x1862023076901888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744185:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076901888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744194:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744196:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744200:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076901888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768104.744205:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768104.744224:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744235:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7c700 x1862023076902016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744246:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076902016/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744255:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744258:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744261:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768104.744268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768104.744287:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7fb80 x1862023076902144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744310:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076902144/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744319:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744322:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744325:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768104.744332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768104.744351:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7ce00 x1862023076902272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076902272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744395:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744398:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744401:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768104.744407:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768104.744428:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744439:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7d880 x1862023076902400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744450:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076902400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744458:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744461:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744465:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768104.744470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768104.744491:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7df80 x1862023076902528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744510:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076902528/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744516:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744522:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768104.744528:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768104.744545:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768104.744555:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768104/real 1775768104] req@ffffa0434aa7e680 x1862023076902656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768104.744566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076902656/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768104.744573:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768104.744576:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768104.744580:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076902656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768104.935147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768104.935190:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768105.767160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768105.767162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768105.959103:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768105.959103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768106.791113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768106.791163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768106.983138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768106.983138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768107.815146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768107.815189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768108.007105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768108.007134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768108.839123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768108.839203:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768109.031184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768109.031184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768109.863156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768109.863173:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863181:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768109.863185:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863199:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4611 00010000:00010000:3.0:1775768109.863208:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768109.863210:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863220:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768109.863266:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076902784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863291:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863294:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768109.863295:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863299:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863302:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863306:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863316:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768109.863322:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076902784:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768109.863328:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076902912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863336:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863339:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768109.863340:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863343:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863347:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863349:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863365:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768109.863375:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076903040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863382:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768109.863386:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863389:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863392:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863395:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768109.863399:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863406:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768109.863415:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076903168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863422:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863425:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768109.863426:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863429:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863431:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863433:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768109.863446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350930700 x1862023076902784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768109.863447:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076903296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863484:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768109.863487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076902784/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768109.863488:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768109.863492:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863500:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863504:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768109.863505:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768109.863508:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768109.863512:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775768109.863513:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768109.863518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076902784:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768109.863525:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076903424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863531:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768109.863535:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863537:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863539:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863541:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768109.863545:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768109.863560:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076903552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863571:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768109.863573:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768109.863575:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768109.863579:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768109.863584:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00080000:1.0:1775768109.863590:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768109.863594:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350932a00 x1862023076902912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768109.863606:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768109.863606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076902912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.863616:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768109.863619:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768109.863620:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.863626:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076902912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768109.863631:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768109.863637:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076903680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768109.863646:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768109.863650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768109.863652:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768109.863654:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768109.863659:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4611 00000100:00100000:3.0:1775768109.863661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350931880 x1862023076903040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768109.863664:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768109.863672:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768109.863677:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768109.863679:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076903040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768109.863708:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076903808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768109.863720:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.863727:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.863732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768109.863740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768109.863766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.863777:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350930e00 x1862023076903168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.863789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076903168/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.863796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.863799:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.863802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768109.863807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768109.863826:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.863836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350932300 x1862023076903296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.863846:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076903296/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.863860:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.863863:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.863866:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076903296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768109.863871:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768109.863888:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.863896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350931c00 x1862023076903424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.863907:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076903424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.863915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.863918:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.863921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768109.863939:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768109.863958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.863968:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350930a80 x1862023076903552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.863979:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076903552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.863987:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.863990:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.864005:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768109.864014:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768109.864049:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.864058:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350930000 x1862023076903680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.864066:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076903680/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.864072:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.864074:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.864076:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768109.864079:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768109.864091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768109.864097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768109/real 1775768109] req@ffffa04350932680 x1862023076903808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768109.864104:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076903808/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768109.864109:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768109.864111:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768109.864113:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076903808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768110.055202:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768110.055207:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768110.887171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768110.887258:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768111.079158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768111.079158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768111.911162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768111.911179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768112.103163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768112.103163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768112.935176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768112.935275:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768113.127177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768113.127177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768113.959127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768113.959183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768114.151136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768114.151136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768114.983176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768114.983281:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768114.983310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768114.983320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983332:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983344:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983352:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768114.983399:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076903936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983427:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983431:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768114.983433:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983437:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983441:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983445:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983455:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768114.983469:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076904064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983503:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768114.983508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983516:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983522:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983528:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983535:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768114.983536:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076903936:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768114.983552:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076904192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983566:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983576:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768114.983578:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983581:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983584:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983588:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983594:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768114.983608:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076904320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983619:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983623:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775768114.983627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768114.983629:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983659:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983666:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983672:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983678:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768114.983689:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076904448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983697:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983699:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768114.983701:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983704:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983708:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983711:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983715:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768114.983723:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076904576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983729:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983732:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768114.983733:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983744:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983746:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983748:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983751:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768114.983759:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076904704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983771:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983774:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768114.983776:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983779:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983780:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983782:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983785:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768114.983794:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076904832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768114.983801:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768114.983804:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768114.983806:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768114.983809:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4616 00000100:00080000:2.0:1775768114.983812:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768114.983814:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768114.983816:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768114.983822:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076904960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768114.984193:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7e300 x1862023076903936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984218:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076903936/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984232:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076903936:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768114.984255:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768114.984284:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984302:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7c000 x1862023076904064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076904064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984325:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984329:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984332:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768114.984359:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768114.984381:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984393:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7f100 x1862023076904192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076904192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984421:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984425:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768114.984434:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768114.984452:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7f480 x1862023076904320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984476:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076904320/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984485:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984491:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768114.984497:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768114.984515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984525:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7d500 x1862023076904448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076904448/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984553:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984556:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984560:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076904448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768114.984565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768114.984583:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7dc00 x1862023076904576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984605:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076904576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984614:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984618:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984621:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768114.984627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768114.984663:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7c380 x1862023076904704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984686:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076904704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984698:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984702:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768114.984708:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768114.984726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7e680 x1862023076904832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076904832/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768114.984768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768114.984785:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768114.984795:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768114/real 1775768114] req@ffffa0434aa7df80 x1862023076904960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768114.984807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076904960/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768114.984815:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768114.984818:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768114.984821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076904960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768115.175148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768115.175186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768116.007128:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768116.007161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768116.199108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768116.199136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768117.031179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768117.031198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768117.223137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768117.223140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768118.055199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768118.055290:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768118.247168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768118.247170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768119.079125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768119.079214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768119.271140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768119.271140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768120.103129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768120.103144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768120.103154:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103164:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4621 00010000:00010000:3.0:1775768120.103164:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768120.103174:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103181:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768120.103218:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076905088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103240:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768120.103242:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103244:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103247:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103250:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768120.103265:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076905216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768120.103272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076905088:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768120.103275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103279:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768120.103281:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103284:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103286:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103288:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103291:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768120.103299:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076905344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103304:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768120.103307:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103309:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103312:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103314:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103317:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768120.103323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076905472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768120.103331:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103335:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4621 00000800:00000100:3.0:1775768120.103337:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768120.103338:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103341:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103344:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768120.103350:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076905600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103356:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768120.103359:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103361:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103363:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103366:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103369:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768120.103375:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076905728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103380:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768120.103383:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103387:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00100000:3.0:1775768120.103389:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350932680 x1862023076905088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768120.103390:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103394:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103399:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768120.103407:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076905856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768120.103413:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103415:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768120.103416:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768120.103420:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076905088/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768120.103421:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103425:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103430:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103435:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768120.103442:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768120.103444:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076905984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768120.103450:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768120.103453:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768120.103456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768120.103457:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076905088:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768120.103458:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768120.103464:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4621 00000100:00080000:1.0:1775768120.103467:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768120.103469:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768120.103472:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768120.103479:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076906112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768120.103483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768120.103509:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350930000 x1862023076905216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103533:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076905216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768120.103554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768120.103574:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103603:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350930a80 x1862023076905344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076905344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103625:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103627:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768120.103636:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768120.103656:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350931c00 x1862023076905472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076905472/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103694:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768120.103704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768120.103721:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350932300 x1862023076905600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076905600/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103761:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076905600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768120.103766:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768120.103784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350930e00 x1862023076905728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103804:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076905728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103815:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768120.103824:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768120.103840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350931880 x1862023076905856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103858:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076905856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103869:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103872:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768120.103877:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768120.103893:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350932a00 x1862023076905984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076905984/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076905984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768120.103945:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076906112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768120.103963:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768120.103972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768120/real 1775768120] req@ffffa04350930700 x1862023076906112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768120.103983:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076906112/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768120.103991:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768120.103994:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768120.103997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076906112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768120.295092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768120.295168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768121.127124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768121.127179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768121.319157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768121.319264:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768122.151136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768122.151143:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768122.343090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768122.343090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768123.175163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768123.175237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768123.367141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768123.367141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768124.199125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768124.199186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768124.391100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768124.391100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768125.223147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768125.223164:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768125.223164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768125.223178:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4626 00010000:00010000:3.0:1775768125.223178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768125.223197:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223205:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768125.223247:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076906240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768125.223270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768125.223274:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223276:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223279:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223282:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223290:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768125.223300:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076906368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768125.223309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076906240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768125.223311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223326:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768125.223328:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223332:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223335:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223339:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768125.223353:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076906496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768125.223359:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223362:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768125.223363:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223366:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223368:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223371:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223375:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768125.223385:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768125.223390:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076906624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768125.223398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768125.223403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223407:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223410:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223413:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223417:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768125.223428:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350930700 x1862023076906240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768125.223432:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076906752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768125.223489:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223491:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768125.223494:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223505:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223509:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768125.223513:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076906240/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768125.223517:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223526:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768125.223549:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768125.223555:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076906880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768125.223555:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223570:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076906240:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768125.223582:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768125.223587:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223591:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223593:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223596:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223600:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768125.223610:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076907008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768125.223611:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768125.223634:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768125.223638:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768125.223640:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768125.223645:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00080000:1.0:1775768125.223651:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768125.223657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768125.223661:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223669:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768125.223680:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350932a00 x1862023076906368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768125.223685:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076907136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768125.223726:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768125.223726:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076906368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223741:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768125.223746:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768125.223746:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768125.223753:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768125.223755:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906368:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768125.223771:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4626 00000100:00100000:3.0:1775768125.223772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906496:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768125.223777:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768125.223782:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768125.223796:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768125.223810:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076907264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768125.223817:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.223831:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350931880 x1862023076906496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.223840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076906496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223846:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.223848:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768125.223855:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768125.223867:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.223872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350930e00 x1862023076906624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.223878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076906624/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.223885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768125.223890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768125.223899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.223903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350932300 x1862023076906752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.223909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076906752/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.223920:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223922:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076906752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768125.223925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076906880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768125.223933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.223938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350931c00 x1862023076906880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.223943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076906880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.223950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076906880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768125.223964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768125.223979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.223984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350930a80 x1862023076907008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.223991:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076907008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.223995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.223997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.223999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768125.224003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768125.224011:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.224016:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350930000 x1862023076907136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.224022:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076907136/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.224042:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.224044:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.224046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768125.224050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768125.224060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768125.224066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768125/real 1775768125] req@ffffa04350932680 x1862023076907264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768125.224072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076907264/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768125.224076:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768125.224079:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768125.224080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076907264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768125.415185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768125.415189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768126.247119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768126.247147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768126.439153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768126.439153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768127.271144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768127.271197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768127.463134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768127.463134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768128.295091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768128.295128:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768128.487095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768128.487095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768129.319134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768129.319258:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768129.511090:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768129.511090:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768130.343155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768130.343169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768130.343311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768130.343320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343329:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343338:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343343:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768130.343374:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076907392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343399:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343403:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768130.343405:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343409:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343412:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343417:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343425:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768130.343430:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076907392:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768130.343438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076907520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343447:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343450:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768130.343452:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343455:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343458:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343461:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343465:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768130.343474:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076907648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343499:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343502:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768130.343503:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343507:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343511:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768130.343512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768130.343518:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343522:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768130.343532:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076907776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768130.343544:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343547:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343550:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343553:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343557:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768130.343561:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7d880 x1862023076907392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768130.343572:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076907904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343580:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768130.343584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076907392/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768130.343585:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768130.343587:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343595:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343598:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343606:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343609:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768130.343609:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343615:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343618:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076907392:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768130.343622:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076908032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768130.343631:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343634:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768130.343636:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768130.343638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768130.343643:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343647:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343650:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343656:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768130.343660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768130.343669:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076908160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768130.343671:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7ce00 x1862023076907520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768130.343683:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343689:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768130.343691:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768130.343699:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343704:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343708:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076907520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768130.343714:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768130.343725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768130.343731:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076908288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768130.343731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907520:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768130.343744:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768130.343747:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768130.343748:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768130.343750:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907648:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768130.343753:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4632 00000100:00080000:2.0:1775768130.343759:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768130.343762:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768130.343765:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768130.343773:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076908416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768130.343777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7fb80 x1862023076907648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.343803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076907648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.343809:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343815:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768130.343820:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768130.343837:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343847:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7c700 x1862023076907776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.343859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076907776/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.343866:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768130.343874:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768130.343886:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7f800 x1862023076907904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.343898:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076907904/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.343907:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343909:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076907904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768130.343913:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768130.343921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7ea00 x1862023076908032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.343932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076908032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.343936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768130.343950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768130.343958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343963:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7ed80 x1862023076908160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.343968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076908160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.343972:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.343974:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.343976:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768130.343979:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768130.343987:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.343991:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7d880 x1862023076908288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.344000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076908288/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.344004:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.344006:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.344008:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768130.344010:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768130.344022:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768130.344049:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768130/real 1775768130] req@ffffa0434aa7df80 x1862023076908416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768130.344061:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076908416/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768130.344068:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768130.344071:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768130.344075:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076908416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768130.535118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768130.535118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768131.367149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768131.367156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768131.559128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768131.559128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768132.391119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768132.391162:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768132.583107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768132.583134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768133.415105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768133.415153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768133.607122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768133.607127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768134.439117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768134.439186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768134.631109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768134.631139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775768135.463131:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463138:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768135.463142:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768135.463145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768135.463149:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4637 00010000:00010000:2.0:1775768135.463154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768135.463158:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463166:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768135.463201:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076908544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463230:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463232:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768135.463234:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463237:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463240:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463244:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463251:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768135.463257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076908544:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768135.463262:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076908672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463269:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463271:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768135.463272:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463274:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463276:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463278:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768135.463294:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076908800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463301:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768135.463303:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463306:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463308:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463310:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463312:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768135.463317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768135.463321:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076908928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463329:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768135.463330:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463332:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463334:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463337:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463339:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768135.463344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076909056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768135.463355:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463361:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00100000:3.0:1775768135.463363:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350932680 x1862023076908544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768135.463364:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463369:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768135.463390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076909184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768135.463390:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076908544/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463408:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768135.463412:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768135.463415:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768135.463420:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768135.463420:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076908544:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768135.463425:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463429:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463432:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463447:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463451:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768135.463457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768135.463461:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076909312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768135.463469:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463472:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768135.463473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768135.463476:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463479:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768135.463480:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463491:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350930000 x1862023076908672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768135.463493:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768135.463502:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768135.463506:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076908672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463513:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463517:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768135.463518:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076909440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768135.463520:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768135.463528:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908800:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768135.463538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768135.463540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768135.463542:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768135.463548:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768135.463549:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4637 00000100:00080000:1.0:1775768135.463555:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768135.463560:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463560:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350930a80 x1862023076908800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463572:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076908800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768135.463574:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768135.463584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768135.463590:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076909568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768135.463592:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768135.463600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768135.463618:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350931c00 x1862023076908928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463631:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076908928/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463649:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463652:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076908928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768135.463657:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768135.463676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350932300 x1862023076909056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076909056/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768135.463708:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768135.463725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463732:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350930e00 x1862023076909184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463738:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076909184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463743:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463744:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463746:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768135.463758:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768135.463767:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463771:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350931880 x1862023076909312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463777:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076909312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463782:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768135.463788:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768135.463796:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463803:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350932a00 x1862023076909440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463814:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076909440/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463821:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768135.463831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768135.463847:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768135.463855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768135/real 1775768135] req@ffffa04350930700 x1862023076909568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768135.463867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076909568/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768135.463874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768135.463877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768135.463879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076909568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768135.655094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768135.655095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768136.487125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768136.487152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768136.679123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768136.679123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768137.511107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768137.511154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768137.703120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768137.703120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768138.535150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768138.535177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768138.727135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768138.727135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768139.559123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768139.559183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768139.751122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768139.751122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768140.583209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768140.583443:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768140.583579:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768140.583590:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583606:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583619:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583625:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768140.583661:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076909696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583690:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583693:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768140.583695:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583698:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583702:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583705:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583713:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768140.583726:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076909824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583735:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583738:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768140.583740:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583743:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583747:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583750:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583755:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768140.583780:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076909952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583789:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583792:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768140.583793:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583796:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583799:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583802:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583806:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768140.583814:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076910080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583822:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583831:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768140.583833:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583836:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583839:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583843:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583847:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768140.583856:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076910208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583864:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583867:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768140.583869:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583872:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583876:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583879:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583883:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768140.583893:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076910336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583899:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583901:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768140.583902:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583904:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583906:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583908:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583911:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768140.583915:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076910464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583920:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583922:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768140.583923:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583925:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00080000:3.0:1775768140.583927:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583929:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583932:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768140.583937:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076910592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768140.583943:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768140.583944:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768140.583945:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768140.583948:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4642 00000100:00100000:0.0:1775768140.583951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076909696:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768140.583952:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768140.583956:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768140.583962:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768140.583971:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076910720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768140.584015:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825ce00 x1862023076909696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076909696/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584093:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584097:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584101:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076909696:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768140.584108:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768140.584128:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584139:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825d880 x1862023076909824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584151:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076909824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584159:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584162:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584165:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768140.584171:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768140.584188:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584196:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825dc00 x1862023076909952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076909952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584213:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584215:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584218:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076909952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768140.584224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768140.584239:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584248:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825d500 x1862023076910080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584258:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076910080/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584264:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768140.584273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768140.584286:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584293:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825e300 x1862023076910208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584303:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076910208/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584310:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584313:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584316:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768140.584321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768140.584335:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825f100 x1862023076910336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076910336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584365:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584368:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584371:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768140.584375:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768140.584411:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584421:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825e680 x1862023076910464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584432:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076910464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584439:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584442:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584445:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768140.584471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768140.584486:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584494:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825c000 x1862023076910592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584502:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076910592/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584513:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584516:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584518:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768140.584522:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768140.584534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768140.584541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768140/real 1775768140] req@ffffa0435825ed80 x1862023076910720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768140.584550:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076910720/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768140.584557:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768140.584559:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768140.584562:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076910720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768140.775128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768140.775148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768141.607109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768141.607142:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768141.799089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768141.799091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768142.631138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768142.631146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768142.823125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768142.823126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768143.655117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768143.655145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768143.847091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768143.847091:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768144.679111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768144.679142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768144.871092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768144.871100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768145.703192:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703202:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768145.703208:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768145.703209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768145.703222:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4647 00010000:00010000:2.0:1775768145.703224:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768145.703233:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703241:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768145.703282:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076910848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768145.703307:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703309:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768145.703311:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703314:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703319:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768145.703328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768145.703352:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076910976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768145.703360:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703365:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768145.703366:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076910848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768145.703367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703379:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703382:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703385:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768145.703390:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768145.703401:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076911104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768145.703411:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703414:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768145.703415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703418:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703420:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703422:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768145.703427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768145.703433:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076911232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768145.703439:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768145.703454:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703457:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768145.703459:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703466:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703472:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703477:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703479:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349642a00 x1862023076910848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768145.703489:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768145.703499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076910848/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768145.703506:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076911360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768145.703524:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768145.703529:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768145.703533:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768145.703535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768145.703538:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768145.703539:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076910848:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768145.703546:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703551:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703555:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768145.703558:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768145.703564:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768145.703569:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076911488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768145.703578:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703581:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768145.703583:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768145.703590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768145.703596:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703601:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703605:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703610:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349643b80 x1862023076910976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768145.703612:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768145.703630:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076911616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768145.703635:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076910976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768145.703642:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768145.703655:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768145.703659:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703674:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:3.0:1775768145.703674:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768145.703686:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768145.703686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.703695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076910976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768145.703699:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703707:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911104:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768145.703709:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768145.703727:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076911744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768145.703737:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768145.703749:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768145.703749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349640380 x1862023076911104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768145.703759:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768145.703760:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768145.703775:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4647 00000100:00080000:0.0:1775768145.703783:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768145.703788:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768145.703791:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768145.703796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076911104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768145.703802:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076911872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768145.703851:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.703858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.703862:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768145.703868:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768145.703891:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349643100 x1862023076911232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.703908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076911232/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.703916:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.703919:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.703921:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768145.703925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768145.703935:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349642300 x1862023076911360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.703950:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076911360/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.703954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.703956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.703957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076911360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768145.703971:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911488:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768145.703980:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.703985:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349642a00 x1862023076911488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.703991:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076911488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.703996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.703997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.703999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768145.704002:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768145.704010:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.704014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349640700 x1862023076911616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.704020:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076911616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.704025:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.704045:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.704047:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768145.704050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768145.704060:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.704066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349642680 x1862023076911744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.704072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076911744/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.704077:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.704078:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.704080:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768145.704084:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768145.704091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768145.704096:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768145/real 1775768145] req@ffffa04349640e00 x1862023076911872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768145.704101:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076911872/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768145.704105:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768145.704107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768145.704109:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076911872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768145.895132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768145.895148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768146.727110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768146.727159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768146.919111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768146.919130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768147.751121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768147.751198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768147.943118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768147.943120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768148.775111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768148.775156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768148.967098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768148.967105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768149.799137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768149.799169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768149.991142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768149.991142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768150.823103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768150.823148:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823158:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768150.823166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823176:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823189:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1775768150.823192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768150.823196:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768150.823236:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076912000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768150.823259:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823263:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768150.823264:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823268:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823273:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823278:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823298:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768150.823309:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076912000:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768150.823316:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076912128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768150.823330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823334:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768150.823336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823341:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823346:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823350:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823355:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768150.823366:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076912256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768150.823373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768150.823381:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768150.823382:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768150.823388:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823392:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823397:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823400:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823406:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768150.823416:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076912384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768150.823424:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823427:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768150.823431:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768150.823432:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349641880 x1862023076912000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768150.823436:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823443:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823456:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823466:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768150.823472:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076912000/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768150.823484:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076912512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768150.823487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.823498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.823506:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076912000:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768150.823508:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823516:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768150.823518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823524:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823529:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823534:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823540:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768150.823541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768150.823564:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076912640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768150.823576:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768150.823583:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823586:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768150.823588:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823593:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00100000:3.0:1775768150.823598:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349641500 x1862023076912128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768150.823601:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823611:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823622:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768150.823629:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076912128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768150.823639:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076912768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768150.823641:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.823655:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.823669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912128:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768150.823670:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768150.823683:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823692:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00100000:3.0:1775768150.823693:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912256:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768150.823698:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768150.823702:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823708:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768150.823719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768150.823724:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076912896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768150.823733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349641c00 x1862023076912256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768150.823751:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768150.823755:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768150.823758:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768150.823761:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4652 00000100:00080000:0.0:1775768150.823765:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768150.823766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076912256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768150.823769:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768150.823780:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768150.823794:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768150.823804:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076913024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768150.823805:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.823822:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768150.823834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768150.823861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.823870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349641180 x1862023076912384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.823881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076912384/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.823888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.823891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.823894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768150.823903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768150.823921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.823931:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349640a80 x1862023076912512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.823943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076912512/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.823954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.823958:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.823961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076912512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768150.823988:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768150.824013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.824025:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349643800 x1862023076912640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.824069:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076912640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.824082:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.824086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.824091:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768150.824101:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768150.824130:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.824202:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349641880 x1862023076912768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.824217:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076912768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.824354:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.824358:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.824362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768150.824370:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768150.824441:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.824459:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349640e00 x1862023076912896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.824471:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076912896/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.824478:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.824481:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.824484:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076912896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768150.824490:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768150.824504:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768150.824511:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768150/real 1775768150] req@ffffa04349642680 x1862023076913024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768150.824521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076913024/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768150.824528:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768150.824531:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768150.824533:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768151.015146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768151.015155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768151.847079:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768151.847154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768152.039128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768152.039133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768152.871086:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768152.871146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768153.063104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768153.063106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768153.895113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768153.895169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768154.087114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768154.087117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768154.919111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768154.919167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768155.111113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768155.111113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768155.943123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768155.943140:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943145:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768155.943148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943156:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943163:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943169:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768155.943174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768155.943203:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076913152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943227:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943231:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768155.943233:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943261:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00100000:3.0:1775768155.943265:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076913152:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768155.943266:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768155.943294:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076913280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943304:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768155.943305:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943308:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943311:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943313:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943315:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768155.943321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768155.943323:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076913408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943330:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943332:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768155.943333:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943336:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943338:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943340:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768155.943351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076913536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943359:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768155.943361:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349640700 x1862023076913152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768155.943364:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768155.943368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943376:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943381:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768155.943384:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076913152/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768155.943388:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943397:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775768155.943398:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943404:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943407:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076913152:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768155.943410:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076913664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943417:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768155.943421:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943424:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00100000:3.0:1775768155.943426:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768155.943428:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943442:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943445:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768155.943450:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768155.943456:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076913792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768155.943464:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349642a00 x1862023076913280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768155.943468:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768155.943470:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943476:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943482:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943485:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943487:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768155.943492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076913920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768155.943500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076913280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768155.943507:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943513:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768155.943514:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768155.943515:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768155.943518:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4657 00000100:00080000:0.0:1775768155.943521:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768155.943521:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913280:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768155.943527:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943532:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768155.943535:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768155.943540:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076914048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768155.943546:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768155.943548:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768155.943550:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768155.943554:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4657 00000800:00000100:3.0:1775768155.943555:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768155.943559:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768155.943563:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768155.943566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768155.943567:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349642300 x1862023076913408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768155.943575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076914176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768155.943578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076913408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943609:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943612:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768155.943621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768155.943641:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349643100 x1862023076913536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943662:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076913536/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943667:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943669:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768155.943674:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768155.943697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349640380 x1862023076913664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943719:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076913664/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943726:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943729:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076913664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768155.943745:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768155.943757:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349643b80 x1862023076913792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076913792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943774:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943776:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768155.943784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768155.943799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349640700 x1862023076913920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943819:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076913920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943830:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943833:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076913920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768155.943838:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768155.943854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349642680 x1862023076914048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943871:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076914048/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768155.943882:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768155.943892:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768155.943896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768155/real 1775768155] req@ffffa04349640e00 x1862023076914176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768155.943902:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076914176/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768155.943906:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768155.943907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768155.943909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768156.135127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768156.135127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768156.967103:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768156.967144:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768157.159145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768157.159281:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768157.991102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768157.991123:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768158.183159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768158.183163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768159.015147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768159.015235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768159.207107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768159.207136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768160.039113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768160.039148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768160.231107:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768160.231107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768161.063112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768161.063117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768161.063119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768161.063124:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063132:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063140:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063146:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768161.063156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768161.063175:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076914304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768161.063207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768161.063211:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063214:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063217:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063220:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063227:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768161.063234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076914304:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768161.063238:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076914432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768161.063246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768161.063249:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063251:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063253:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063255:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768161.063267:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076914560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768161.063273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768161.063277:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063279:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063281:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063283:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768161.063288:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063289:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768161.063297:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076914688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768161.063303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768161.063306:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063309:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063311:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063313:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063316:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768161.063320:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349641880 x1862023076914304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768161.063323:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076914816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768161.063343:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063349:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768161.063352:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063355:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063357:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063358:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063363:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768161.063369:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076914944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768161.063375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076914304/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768161.063376:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063382:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768161.063383:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063389:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063393:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063396:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768161.063397:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768161.063398:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768161.063406:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768161.063408:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076915072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768161.063412:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076914304:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768161.063416:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063421:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768161.063422:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063426:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063428:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063430:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063432:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768161.063438:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076915200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768161.063442:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914432:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768161.063450:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768161.063452:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768161.063454:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768161.063457:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4662 00000100:00080000:0.0:1775768161.063461:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768161.063463:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768161.063466:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768161.063468:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768161.063476:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076915328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768161.063482:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349643800 x1862023076914432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063492:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076914432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063498:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063501:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768161.063508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768161.063519:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063524:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349640a80 x1862023076914560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076914560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063539:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063540:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063542:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768161.063545:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768161.063554:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063561:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349641180 x1862023076914688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063571:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076914688/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063578:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063584:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768161.063589:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768161.063606:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063615:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349641c00 x1862023076914816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063622:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076914816/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063627:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063631:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076914816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768161.063635:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768161.063647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063652:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349641500 x1862023076914944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076914944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063663:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063664:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063666:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076914944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768161.063669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768161.063677:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349640e00 x1862023076915072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076915072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063692:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063695:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768161.063709:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768161.063717:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063722:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349642680 x1862023076915200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076915200/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063738:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768161.063744:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768161.063752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768161.063756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768161/real 1775768161] req@ffffa04349640700 x1862023076915328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768161.063762:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076915328/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768161.063767:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768161.063768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768161.063769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076915328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768161.255096:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768161.255133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768162.087164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768162.087223:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768162.279165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768162.279170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768163.111137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768163.111201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768163.303258:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768163.303264:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768164.135200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768164.135281:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768164.327189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768164.327189:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768165.159185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768165.159242:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768165.351197:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768165.351197:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768166.183125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768166.183148:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183154:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1775768166.183162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768166.183163:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183196:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768166.183232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076915456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183255:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768166.183261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183273:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183282:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768166.183282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076915456:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768166.183296:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076915584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183303:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183305:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768166.183307:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183310:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183315:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183318:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183323:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768166.183335:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076915712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183343:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183347:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768166.183349:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183354:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183358:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183362:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768166.183364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915584:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775768166.183370:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768166.183386:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076915840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768166.183391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768166.183403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183406:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768166.183408:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768166.183412:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915840:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768166.183415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183421:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183425:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768166.183442:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076915968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183453:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183455:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768166.183457:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183460:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183464:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768166.183466:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915968:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768166.183470:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183474:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768166.183483:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076916096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183491:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183494:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768166.183495:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768166.183497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183503:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183505:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183508:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183512:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768166.183519:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076916224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183528:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768166.183534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00100000:3.0:1775768166.183541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768166.183543:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768166.183550:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183555:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768166.183566:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076916352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768166.183581:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768166.183584:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768166.183586:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768166.183592:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4667 00000100:00080000:1.0:1775768166.183599:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768166.183602:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916352:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768166.183609:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768166.183615:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768166.183627:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076916480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768166.183661:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768166.375120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768166.375120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768167.207166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768167.207199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768167.399114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768167.399114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768168.231166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768168.231195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768168.423124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768168.423124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768169.255130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768169.255141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1775768169.255304:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775768169.255339:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768169.255353:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255401:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255415:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255424:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255448:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255455:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255461:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255468:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768169.255475:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775768169.255539:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350932680 x1862023076916480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255564:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076916480/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255585:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768169.255592:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350930000 x1862023076916352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255604:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076916352/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255612:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255615:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768169.255626:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350930a80 x1862023076916224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076916224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255646:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255650:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255653:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768169.255658:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350931c00 x1862023076916096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076916096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255676:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255680:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255683:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076916096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768169.255689:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350932300 x1862023076915968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255697:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076915968/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255705:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768169.255717:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350930e00 x1862023076915840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076915840/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255739:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255742:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768169.255747:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350932a00 x1862023076915584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255757:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076915584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768169.255775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350931880 x1862023076915712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076915712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255796:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076915712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768169.255802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768166/real 1775768169] req@ffffa04350930700 x1862023076915456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768169.255881:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076915456/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768169.255891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768169.255894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768169.255897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076915456:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768169.447134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768169.447134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768170.279122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768170.279171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768170.471106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768170.471112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768171.303139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768171.303173:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303178:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768171.303181:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1775768171.303185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768171.303189:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303198:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768171.303203:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768171.303232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076916608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768171.303257:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303260:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768171.303262:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303268:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768171.303272:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303283:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768171.303299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076916608:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768171.303304:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076916736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768171.303321:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303324:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768171.303326:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303330:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303339:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768171.303343:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303347:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768171.303360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076916864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768171.303370:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768171.303372:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303376:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768171.303378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303383:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303388:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768171.303394:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303399:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768171.303411:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076916992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768171.303417:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349643b80 x1862023076916608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768171.303424:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303433:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768171.303435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303441:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303446:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768171.303446:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076916608/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768171.303453:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303465:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775768171.303466:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.303475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.303479:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076916608:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768171.303486:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076917120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768171.303496:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303498:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768171.303500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303504:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303510:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768171.303514:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768171.303516:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768171.303537:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076917248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768171.303548:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768171.303553:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768171.303554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768171.303557:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768171.303562:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303567:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768171.303572:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349640380 x1862023076916736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768171.303575:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303583:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768171.303593:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076916736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768171.303597:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076917376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768171.303614:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768171.303618:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768171.303621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768171.303623:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768171.303626:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768171.303628:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916736:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768171.303632:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303638:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768171.303640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768171.303645:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768171.303666:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076917504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768171.303672:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768171.303681:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768171.303686:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768171.303689:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768171.303690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349643100 x1862023076916864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768171.303697:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4672 00000100:00080000:0.0:1775768171.303705:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768171.303709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076916864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768171.303710:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768171.303717:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768171.303725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.303730:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768171.303734:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076917632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768171.303735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768171.303747:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768171.303773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.303784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349642300 x1862023076916992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.303797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076916992/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.303804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.303807:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.303811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076916992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768171.303816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076917120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768171.303835:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.303928:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349642a00 x1862023076917120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.303939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076917120/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.303950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.303954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.303956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076917120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768171.303974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768171.303993:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.304003:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349643b80 x1862023076917248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.304014:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076917248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.304048:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.304051:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.304054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768171.304060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768171.304079:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.304103:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349640700 x1862023076917376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.304114:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076917376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.304121:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.304124:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.304126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768171.304131:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768171.304151:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.304161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349642680 x1862023076917504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.304173:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076917504/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.304179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.304182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.304185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768171.304190:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768171.304206:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768171.304216:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768171/real 1775768171] req@ffffa04349640e00 x1862023076917632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768171.304227:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076917632/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768171.304234:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768171.304236:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768171.304238:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076917632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768171.495105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768171.495114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768172.327146:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768172.327191:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768172.519112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768172.519113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768173.351135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768173.351168:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768173.543132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768173.543132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768174.375147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768174.375216:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768174.567140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768174.567140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768175.399126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768175.399165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768175.591140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768175.591140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768176.423108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768176.423118:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423123:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768176.423126:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423134:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423143:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768176.423149:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768176.423166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768176.423181:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076917760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423203:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768176.423205:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423209:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423212:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768176.423216:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423225:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768176.423236:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076917760:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768176.423241:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076917888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423252:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423254:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768176.423256:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423258:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423261:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768176.423264:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423268:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768176.423276:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076918016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423284:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768176.423285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423288:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423291:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768176.423294:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768176.423299:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423301:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768176.423313:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076918144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423325:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768176.423327:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423331:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423335:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768176.423340:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349641500 x1862023076917760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768176.423341:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768176.423359:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076918272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768176.423365:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076917760/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768176.423374:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768176.423377:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768176.423378:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768176.423379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768176.423385:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768176.423388:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423392:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768176.423394:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076917760:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768176.423397:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423403:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768176.423410:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076918400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423418:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768176.423419:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423421:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423423:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768176.423424:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076917888:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768176.423427:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423431:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768176.423438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076918528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423444:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768176.423446:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768176.423447:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423450:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423452:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768176.423456:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768176.423457:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768176.423469:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076918656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768176.423475:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768176.423477:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349641c00 x1862023076917888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768176.423478:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768176.423480:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768176.423486:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4678 00000100:00080000:0.0:1775768176.423491:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768176.423497:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768176.423501:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768176.423502:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076917888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768176.423512:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076918784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768176.423528:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423533:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423537:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076917888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768176.423544:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768176.423568:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423581:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349641180 x1862023076918016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423593:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076918016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423601:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423604:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423607:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768176.423612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768176.423630:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349640a80 x1862023076918144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423650:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076918144/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423666:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423669:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768176.423678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768176.423697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349643800 x1862023076918272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076918272/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423726:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423729:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076918272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768176.423738:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768176.423753:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349641880 x1862023076918400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076918400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768176.423811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768176.423828:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349641500 x1862023076918528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423848:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076918528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768176.423865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768176.423880:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349640e00 x1862023076918656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076918656/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768176.423922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768176.423939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768176.423948:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768176/real 1775768176] req@ffffa04349642680 x1862023076918784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768176.423958:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076918784/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768176.423965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768176.423967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768176.423971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076918784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768176.615156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768176.615156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768177.447148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768177.447242:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768177.639156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768177.639166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768178.471120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768178.471182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768178.663141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768178.663141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768179.495092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768179.495215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768179.687151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768179.687151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768180.519120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768180.519151:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768180.711125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768180.711125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768181.543165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768181.543185:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543199:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543214:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543223:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768181.543274:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076918912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768181.543312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543316:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768181.543319:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543324:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543328:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543332:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543343:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768181.543356:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076919040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768181.543366:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543370:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768181.543373:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543378:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543383:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543387:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543392:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768181.543394:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076918912:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768181.543411:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076919168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768181.543422:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543426:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768181.543428:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543432:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543438:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543442:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543448:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768181.543459:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076919296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768181.543469:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768181.543474:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768181.543477:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768181.543480:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543486:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543491:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543495:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543500:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768181.543512:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076919424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768181.543523:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543528:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775768181.543529:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349640700 x1862023076918912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768181.543532:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543544:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543551:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543557:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543562:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768181.543564:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076918912/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.543575:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768181.543579:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076919552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768181.543582:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.543588:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076918912:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768181.543600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919040:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768181.543616:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543620:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768181.543623:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768181.543625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768181.543633:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543638:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768181.543640:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349643b80 x1862023076919040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768181.543647:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543657:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768181.543658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076919040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.543671:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768181.543680:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076919680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768181.543683:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.543694:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919040:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768181.543696:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543700:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768181.543704:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543719:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00080000:0.0:1775768181.543726:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768181.543730:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919168:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768181.543735:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543743:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00010000:00010000:2.0:1775768181.543755:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768181.543762:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076919808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768181.543770:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768181.543772:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768181.543790:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768181.543795:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768181.543799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349642a00 x1862023076919168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768181.543808:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768181.543819:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4683 00000100:00100000:3.0:1775768181.543822:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076919168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768181.543826:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768181.543835:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768181.543844:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768181.543848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.543854:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768181.543860:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076919936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768181.543870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768181.543880:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768181.543902:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.543912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349642300 x1862023076919296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.543922:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076919296/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.543929:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.543934:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.543937:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768181.543943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768181.543957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.543965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349643100 x1862023076919424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.543972:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076919424/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.543978:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.543979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.543982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076919424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768181.543986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768181.544015:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.544023:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349640380 x1862023076919552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.544052:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076919552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.544058:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.544061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.544063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768181.544067:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768181.544081:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.544088:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349642680 x1862023076919680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.544095:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076919680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.544100:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.544102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.544104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768181.544120:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768181.544131:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.544138:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349640e00 x1862023076919808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.544145:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076919808/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.544152:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.544155:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.544158:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768181.544164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768181.544184:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768181.544195:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768181/real 1775768181] req@ffffa04349641500 x1862023076919936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768181.544208:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076919936/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768181.544217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768181.544221:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768181.544224:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076919936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768181.735192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768181.735197:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768182.567134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768182.567184:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768182.759106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768182.759128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768183.591129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768183.591188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768183.783123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768183.783123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768184.615128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768184.615168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768184.807172:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768184.807172:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768185.639211:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768185.639295:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768185.831181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768185.831181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768186.663113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768186.663126:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663135:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663145:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663152:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768186.663160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768186.663196:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076920064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1775768186.663197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768186.663226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768186.663232:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663237:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663241:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663244:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768186.663262:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076920192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768186.663272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768186.663279:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663285:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663292:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663296:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768186.663319:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076920320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768186.663322:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076920064:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768186.663333:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768186.663337:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663342:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663346:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663354:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663359:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768186.663370:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076920448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768186.663377:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1775768186.663381:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768186.663384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768186.663386:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663395:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663404:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663409:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663416:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768186.663426:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349641880 x1862023076920064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768186.663430:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076920576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768186.663459:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663461:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768186.663463:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768186.663468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076920064/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768186.663472:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663478:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768186.663481:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768186.663484:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663489:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1775768186.663489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076920064:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768186.663501:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768186.663503:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076920704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768186.663511:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663513:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768186.663514:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663517:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663519:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663521:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663524:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1775768186.663527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768186.663535:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076920832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768186.663541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349643800 x1862023076920192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768186.663546:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768186.663555:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663559:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663562:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775768186.663562:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076920192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768186.663568:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768186.663574:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1775768186.663576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768186.663589:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076920960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768186.663596:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768186.663598:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768186.663598:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768186.663601:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4688 00000100:00080000:0.0:1775768186.663603:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768186.663606:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920320:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768186.663611:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768186.663622:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076921088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775768186.663639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663651:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349640a80 x1862023076920320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663661:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076920320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663670:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663673:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768186.663683:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768186.663706:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663720:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349641180 x1862023076920448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076920448/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663738:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663748:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663750:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768186.663759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768186.663776:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663793:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349641c00 x1862023076920576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076920576/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076920576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768186.663834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768186.663855:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349641500 x1862023076920704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076920704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768186.663903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768186.663920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349640e00 x1862023076920832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076920832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663952:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663956:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768186.663960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768186.663973:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.663979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349642680 x1862023076920960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.663985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076920960/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.663993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.663995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.663997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076920960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768186.664015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768186.664073:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768186.664085:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768186/real 1775768186] req@ffffa04349640380 x1862023076921088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768186.664099:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076921088/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768186.664109:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768186.664112:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768186.664114:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768186.855162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768186.855167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768187.687192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768187.687227:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768187.879121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768187.879239:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768188.711071:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768188.711129:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768188.903122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768188.903144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768189.735125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768189.735169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768189.927087:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768189.927122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768190.759179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768190.759231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768190.951158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768190.951195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768191.783130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768191.783154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768191.783164:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783172:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768191.783177:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768191.783188:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783200:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783206:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768191.783241:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076921216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768191.783267:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783270:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768191.783272:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768191.783277:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783281:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783285:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076921216:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768191.783296:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768191.783311:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076921344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768191.783332:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783336:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768191.783338:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768191.783341:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783345:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783349:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768191.783353:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768191.783356:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768191.783369:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076921472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768191.783379:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768191.783385:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768191.783389:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00100000:3.0:1775768191.783392:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7e680 x1862023076921216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783394:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783403:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768191.783411:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768191.783418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076921216/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768191.783425:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076921600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768191.783428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783434:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783440:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076921216:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768191.783445:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783458:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768191.783460:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768191.783460:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768191.783469:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783478:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768191.783483:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775768191.783488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768191.783497:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076921728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768191.783500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7c380 x1862023076921344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783525:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768191.783525:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076921344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768191.783538:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768191.783539:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768191.783546:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00100000:3.0:1775768191.783551:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921344:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768191.783552:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768191.783556:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768191.783559:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768191.783572:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768191.783576:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7dc00 x1862023076921472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768191.783588:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076921856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768191.783602:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076921472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768191.783631:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768191.783632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783641:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768191.783647:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921600:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768191.783650:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768191.783653:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768191.783658:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4693 00000800:00000100:3.0:1775768191.783662:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768191.783664:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768191.783670:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783670:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7d500 x1862023076921600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768191.783678:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768191.783681:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076921600/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783689:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768191.783693:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076921984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768191.783693:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768191.783706:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921728:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768191.783708:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768191.783712:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768191.783715:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768191.783720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768191.783721:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783727:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768191.783730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7f480 x1862023076921728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783732:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076921728/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783748:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768191.783749:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768191.783752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783760:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076921728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768191.783765:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076922112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768191.783768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768191.783782:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768191.783786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768191.783789:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768191.783792:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768191.783796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7f100 x1862023076921856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783800:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4693 00000100:00080000:2.0:1775768191.783807:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768191.783808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076921856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768191.783815:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768191.783817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783821:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768191.783822:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768191.783824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768191.783835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768191.783839:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076922240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768191.783853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783860:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7c000 x1862023076921984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768191.783867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076921984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783876:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076921984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768191.783879:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076922112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768191.783898:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783907:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7e300 x1862023076922112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768191.783915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076922112/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076922112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768191.783938:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076922240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768191.783955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768191.783964:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768191/real 1775768191] req@ffffa0434aa7d500 x1862023076922240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768191.783974:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076922240/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768191.783981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768191.783983:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768191.783986:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076922240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768191.975179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768191.975206:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768192.807126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768192.807166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768192.999121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768192.999141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768193.831106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768193.831133:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768194.023108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768194.023114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768194.855115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768194.855150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768195.047115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768195.047118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768195.879106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768195.879141:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768196.071123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768196.071123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768196.903132:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768196.903135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768196.903141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768196.903145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903154:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4698 00010000:00010000:3.0:1775768196.903168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768196.903171:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903178:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768196.903212:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076922368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903232:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903235:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768196.903236:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903239:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903241:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903244:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903251:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768196.903260:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076922496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768196.903266:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076922368:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768196.903269:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903276:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768196.903277:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903279:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903281:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903283:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768196.903293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076922624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903301:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768196.903303:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903304:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903316:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903320:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768196.903329:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768196.903336:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076922752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903345:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903348:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768196.903351:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903355:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903359:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903363:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903369:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768196.903370:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7dc00 x1862023076922368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768196.903379:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076922880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903389:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903391:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768196.903393:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903396:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00100000:3.0:1775768196.903399:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076922368/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768196.903400:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903405:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903411:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768196.903420:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768196.903422:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076923008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768196.903429:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768196.903431:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903435:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768196.903437:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768196.903438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076922368:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768196.903442:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903446:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903449:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903452:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768196.903457:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076923136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768196.903465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768196.903468:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922496:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768196.903470:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903474:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903478:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768196.903480:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768196.903487:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076923264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768196.903496:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768196.903499:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1775768196.903502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768196.903503:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768196.903510:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4698 00000100:00080000:2.0:1775768196.903514:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768196.903518:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903521:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7c380 x1862023076922496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768196.903524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768196.903541:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076923392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768196.903542:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076922496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903573:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903577:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768196.903583:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768196.903601:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903610:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7e680 x1862023076922624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903621:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076922624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903628:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903631:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903634:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768196.903640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768196.903657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903667:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7df80 x1862023076922752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076922752/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903687:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768196.903696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768196.903711:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903720:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7d880 x1862023076922880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903731:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076922880/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903739:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903742:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903745:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076922880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768196.903750:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768196.903766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7ed80 x1862023076923008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903786:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076923008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903793:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903798:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768196.903803:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768196.903819:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7ea00 x1862023076923136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903868:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076923136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903880:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768196.903897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768196.903916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7f800 x1862023076923264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076923264/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903947:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903949:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768196.903952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768196.903965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768196.903970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768196/real 1775768196] req@ffffa0434aa7c700 x1862023076923392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768196.903976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076923392/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768196.903981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768196.903983:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768196.903985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076923392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768197.095128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768197.095128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768197.927171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768197.927214:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768198.119105:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768198.119151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768198.951166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768198.951228:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768199.143129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768199.143133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768199.975124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768199.975197:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768200.167103:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768200.167113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768200.999145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768200.999151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768201.191106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768201.191106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768202.023122:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023129:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768202.023134:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023141:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023150:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768202.023153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768202.023155:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768202.023170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768202.023186:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076923520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023205:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023207:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768202.023208:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023211:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023213:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023216:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023223:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768202.023237:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076923520:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768202.023238:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076923648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023250:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023252:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768202.023253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023255:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023258:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023260:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023263:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768202.023271:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076923776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023277:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023279:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768202.023280:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023283:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023285:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023288:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023292:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768202.023300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076923904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768202.023302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768202.023311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023313:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768202.023314:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023317:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023320:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023322:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768202.023332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076924032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023340:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768202.023341:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768202.023345:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7fb80 x1862023076923520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768202.023346:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023350:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023354:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023360:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768202.023369:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076924160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768202.023374:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076923520/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768202.023378:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023384:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768202.023387:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023392:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023395:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768202.023395:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768202.023399:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023402:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768202.023403:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023408:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076923520:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768202.023413:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076924288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023421:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768202.023422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023424:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023426:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768202.023432:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023432:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923648:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768202.023435:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768202.023443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076924416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768202.023450:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768202.023452:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768202.023453:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768202.023456:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4703 00000100:00080000:2.0:1775768202.023459:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768202.023462:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768202.023464:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768202.023468:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768202.023475:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076924544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768202.023481:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7ce00 x1862023076923648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023495:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076923648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023503:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023507:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023510:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768202.023515:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768202.023530:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7c700 x1862023076923776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023547:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076923776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023557:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023560:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768202.023565:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768202.023579:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023587:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7f800 x1862023076923904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023596:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076923904/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023603:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023606:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023609:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076923904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768202.023614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768202.023627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023635:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7ea00 x1862023076924032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023644:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076924032/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023653:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023656:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768202.023661:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924160:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768202.023674:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7ed80 x1862023076924160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076924160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023703:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768202.023708:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768202.023722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023730:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7d880 x1862023076924288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023739:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076924288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023746:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023748:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023751:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768202.023768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768202.023784:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023808:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7df80 x1862023076924416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023819:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076924416/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023826:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023829:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023832:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768202.023837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768202.023854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768202.023862:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768202/real 1775768202] req@ffffa0434aa7e680 x1862023076924544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768202.023872:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076924544/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768202.023879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768202.023882:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768202.023884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076924544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768202.215088:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768202.215092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768203.047119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768203.047147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768203.239101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768203.239101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768204.071109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768204.071186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768204.263114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768204.263114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768205.095133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768205.095163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768205.287129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768205.287129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768206.119137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768206.119137:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768206.311129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768206.311146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768207.143126:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143136:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768207.143141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143154:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00100000:0.0:1775768207.143158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768207.143158:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768207.143173:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143181:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768207.143219:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076924672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143243:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768207.143248:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143254:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143258:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143262:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143272:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768207.143281:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076924672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768207.143288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076924800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143299:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143302:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768207.143304:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143308:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143311:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143315:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143319:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768207.143330:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076924928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143340:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768207.143342:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768207.143347:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768207.143350:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143355:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143359:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143363:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143369:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768207.143381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076925056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143391:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143393:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768207.143396:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768207.143398:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7c380 x1862023076924672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768207.143402:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143408:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143414:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143421:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768207.143431:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076924672/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768207.143432:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076925184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143464:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143467:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768207.143468:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768207.143469:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143475:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143477:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768207.143480:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768207.143483:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143488:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768207.143490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076924672:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768207.143497:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076925312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143504:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143506:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768207.143508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143510:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143512:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143514:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768207.143519:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768207.143526:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076925440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143533:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768207.143537:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143540:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143543:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143546:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143550:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768207.143550:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768207.143561:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076925568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768207.143568:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768207.143570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768207.143570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7dc00 x1862023076924800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768207.143573:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768207.143579:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4708 00000100:00080000:2.0:1775768207.143582:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768207.143588:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768207.143605:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768207.143609:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076924800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768207.143618:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076925696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768207.143647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.143653:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.143657:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768207.143672:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768207.143691:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.143702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7d500 x1862023076924928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.143714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076924928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.143730:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.143734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.143737:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076924928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768207.143757:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768207.143777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.143796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7e300 x1862023076925056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.143809:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076925056/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.143817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.143820:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.143824:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768207.143830:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768207.143849:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.143869:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7c000 x1862023076925184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.143882:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076925184/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.143891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.143894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.143897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768207.143903:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925312:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768207.143920:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.143930:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7f100 x1862023076925312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.143942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076925312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.143951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.143954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.143957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768207.143974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768207.143991:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.144001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7f480 x1862023076925440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.144012:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076925440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.144020:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.144023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.144046:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768207.144052:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768207.144069:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.144079:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7c380 x1862023076925568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.144090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076925568/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.144098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.144101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.144104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768207.144110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768207.144125:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768207.144134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768207/real 1775768207] req@ffffa0434aa7e680 x1862023076925696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768207.144145:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076925696/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768207.144154:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768207.144157:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768207.144160:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076925696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768207.335147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768207.335196:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768208.167108:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768208.167191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768208.359175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768208.359175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768209.191132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768209.191162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768209.383104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768209.383113:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768210.215136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768210.215173:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768210.407135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768210.407138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768211.239130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768211.239155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768211.431078:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768211.431089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768212.263118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768212.263123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768212.263126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768212.263131:0:8:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263139:0:8:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263149:0:8:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263156:0:8:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768212.263167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768212.263230:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076925824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768212.263249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768212.263254:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263263:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263269:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263273:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263282:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768212.263283:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076925824:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768212.263298:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076925952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768212.263310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768212.263315:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263319:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263323:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263327:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263332:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768212.263344:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076926080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768212.263354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768212.263354:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768212.263359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768212.263361:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263366:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263370:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263374:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263378:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768212.263389:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076926208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768212.263398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263401:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768212.263403:0:8:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768212.263405:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349643100 x1862023076925824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768212.263409:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263416:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263422:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263427:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768212.263431:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076925824/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263442:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263448:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768212.263451:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076926336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768212.263455:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076925824:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768212.263468:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263470:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768212.263471:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263474:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263477:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263480:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263483:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768212.263489:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768212.263491:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076926464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768212.263501:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263508:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768212.263510:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263512:0:8:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263515:0:8:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768212.263519:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768212.263521:0:8:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263526:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768212.263533:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076926592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768212.263539:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349642300 x1862023076925952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768212.263540:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768212.263546:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768212.263549:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768212.263558:0:8:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263562:0:8:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263565:0:8:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263570:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768212.263572:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076925952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768212.263582:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076926720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768212.263584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263591:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768212.263600:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768212.263600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076925952:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768212.263607:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768212.263609:0:8:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768212.263612:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926080:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768212.263616:0:8:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4713 00000100:00080000:0.0:1775768212.263621:0:8:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768212.263625:0:8:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768212.263631:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768212.263644:0:8:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076926848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768212.263651:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349642a00 x1862023076926080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263681:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076926080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263703:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.263710:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768212.263716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768212.263735:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349643b80 x1862023076926208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076926208/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263763:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.263769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768212.263775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768212.263792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349640700 x1862023076926336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076926336/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263819:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.263825:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076926336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768212.263843:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768212.263861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263870:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349643100 x1862023076926464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263880:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076926464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263888:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.263894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768212.263899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768212.263915:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263924:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349640380 x1862023076926592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263935:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076926592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.263947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768212.263952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768212.263969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.263978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349642680 x1862023076926720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.263988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076926720/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.263995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.263998:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.264001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768212.264007:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768212.264023:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768212.264053:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768212/real 1775768212] req@ffffa04349640e00 x1862023076926848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768212.264064:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076926848/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768212.264072:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768212.264075:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768212.264078:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076926848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768212.455131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768212.455137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768213.287117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768213.287122:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768213.479118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768213.479118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768214.311092:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768214.311111:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768214.503084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768214.503086:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768215.335105:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768215.335113:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768215.527123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768215.527124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768216.359126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768216.359152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768216.551147:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768216.551148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768217.383117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768217.383143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768217.383150:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383157:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768217.383161:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383171:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383182:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383188:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768217.383246:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076926976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383282:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768217.383284:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383292:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383298:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383308:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768217.383322:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076926976:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768217.383324:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076927104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383342:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768217.383344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383347:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383350:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768217.383367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076927232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383374:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383376:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768217.383378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383379:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383381:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383384:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383386:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768217.383393:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076927360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768217.383403:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768217.383405:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768217.383408:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383411:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383413:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768217.383423:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076927488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383429:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383431:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768217.383432:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383434:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383442:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383444:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383447:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768217.383453:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076927616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383461:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768217.383462:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383466:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383468:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768217.383468:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350930700 x1862023076926976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768217.383474:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383479:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768217.383489:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076927744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383495:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383499:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768217.383499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076926976/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768217.383500:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768217.383506:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383509:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383514:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383519:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768217.383520:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768217.383527:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076927872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768217.383532:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768217.383534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768217.383535:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768217.383537:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768217.383544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076926976:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768217.383545:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4719 00000100:00080000:1.0:1775768217.383553:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768217.383558:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768217.383563:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768217.383574:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076928000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768217.383576:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768217.383606:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383622:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350931880 x1862023076927104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076927104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383654:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383661:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768217.383666:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768217.383686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350932a00 x1862023076927232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076927232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383715:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768217.383726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768217.383743:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350930e00 x1862023076927360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383764:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076927360/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768217.383782:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768217.383799:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350932300 x1862023076927488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383828:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076927488/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383836:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383839:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076927488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768217.383847:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927616:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768217.383864:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350931c00 x1862023076927616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076927616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768217.383902:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768217.383919:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383928:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350930a80 x1862023076927744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076927744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.383946:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.383949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.383952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768217.383957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768217.383972:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.383982:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350930000 x1862023076927872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.383994:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076927872/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.384002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.384004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.384008:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076927872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768217.384044:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768217.384062:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768217.384072:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768217/real 1775768217] req@ffffa04350932680 x1862023076928000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768217.384081:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076928000/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768217.384088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768217.384091:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768217.384094:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768217.575156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768217.575156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768218.407133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768218.407150:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768218.599157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768218.599157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768219.431122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768219.431163:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768219.623137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768219.623221:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768220.455155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768220.455195:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768220.647184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768220.647312:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768221.479141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768221.479167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768221.671089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768221.671089:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768222.503134:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503143:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768222.503148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503158:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503169:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503176:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768222.503181:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768222.503212:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076928128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768222.503221:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768222.503235:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768222.503241:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503247:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503254:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503259:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768222.503269:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768222.503283:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076928256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768222.503295:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076928128:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768222.503296:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768222.503310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503315:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503320:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503325:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768222.503330:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768222.503344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076928384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768222.503361:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503365:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768222.503367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503372:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503376:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503380:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768222.503387:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1775768222.503390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768222.503408:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076928512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768222.503421:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503424:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768222.503426:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503430:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503435:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503439:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768222.503444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768222.503460:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350932680 x1862023076928128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768222.503461:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076928640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768222.503516:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503523:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768222.503525:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503532:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00100000:0.0:1775768222.503537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076928128/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768222.503542:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503555:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768222.503560:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768222.503561:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768222.503570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768222.503579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076928768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768222.503586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076928128:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768222.503619:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503622:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768222.503624:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503630:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503635:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503639:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768222.503646:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768222.503650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768222.503663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076928896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768222.503675:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503679:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768222.503680:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768222.503686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768222.503687:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503695:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503709:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768222.503716:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350930000 x1862023076928256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768222.503717:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768222.503737:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076929024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768222.503750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076928256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768222.503750:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768222.503763:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768222.503767:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768222.503777:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4724 00000100:00080000:1.0:1775768222.503787:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768222.503795:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768222.503798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768222.503799:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775768222.503809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.503818:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768222.503818:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076929152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768222.503840:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768222.503870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.503882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350930a80 x1862023076928384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.503895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076928384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.503903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.503906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.503910:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768222.503923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768222.503944:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.503955:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350931c00 x1862023076928512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.503967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076928512/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.503977:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.503981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.503985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768222.503991:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768222.504010:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.504021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350932300 x1862023076928640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.504085:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076928640/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.504094:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.504098:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.504101:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076928640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768222.504108:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768222.504127:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.504138:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350930e00 x1862023076928768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.504150:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076928768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.504159:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.504164:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.504168:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768222.504192:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928896:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768222.504211:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.504221:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350932a00 x1862023076928896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.504233:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076928896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.504242:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.504245:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.504249:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076928896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768222.504254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768222.504271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.504281:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350931880 x1862023076929024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.504293:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076929024/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.504301:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.504304:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.504308:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768222.504314:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768222.504332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768222.504343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768222/real 1775768222] req@ffffa04350930700 x1862023076929152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768222.504354:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076929152/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768222.504363:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768222.504366:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768222.504369:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768222.695155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768222.695155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768223.527141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768223.527152:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768223.719091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768223.719096:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768224.551117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768224.551142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768224.743136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768224.743136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768225.575111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768225.575133:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768225.767134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768225.767134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768226.599141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768226.599155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768226.791137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768226.791137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768227.623115:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623124:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768227.623129:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768227.623136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768227.623139:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4729 00010000:00010000:3.0:1775768227.623148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768227.623162:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623169:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768227.623209:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076929280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768227.623230:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623233:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768227.623235:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623239:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623242:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623246:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768227.623255:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768227.623265:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076929280:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768227.623274:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076929408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768227.623283:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623289:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768227.623290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623308:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623312:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768227.623316:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775768227.623317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768227.623325:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076929536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768227.623332:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768227.623336:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623339:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623341:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623344:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768227.623348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768227.623351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350930700 x1862023076929280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768227.623356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076929664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768227.623363:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623366:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768227.623367:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076929280/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768227.623368:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623371:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623374:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768227.623376:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768227.623377:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768227.623379:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:1.0:1775768227.623380:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768227.623383:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076929280:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768227.623387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076929792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768227.623393:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623396:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768227.623398:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623401:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623404:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623407:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623411:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929408:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775768227.623415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768227.623423:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076929920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768227.623430:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768227.623432:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623435:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768227.623437:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768227.623439:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350931880 x1862023076929408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768227.623440:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00080000:1.0:1775768227.623443:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623446:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623450:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076929408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768227.623450:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768227.623458:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768227.623458:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076930048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768227.623461:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623464:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768227.623469:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768227.623473:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768227.623479:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768227.623480:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768227.623483:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4729 00000800:00000100:0.0:1775768227.623484:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768227.623486:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623490:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623493:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350932a00 x1862023076929536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768227.623494:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768227.623502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076930176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768227.623503:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076929536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623511:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768227.623512:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768227.623514:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768227.623515:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768227.623516:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768227.623517:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768227.623520:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4729 00000100:00100000:0.0:1775768227.623522:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929664:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768227.623522:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768227.623525:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768227.623529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768227.623536:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076930304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768227.623537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623569:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350930e00 x1862023076929664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623581:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076929664/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623589:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623595:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768227.623600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768227.623616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350932300 x1862023076929792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623630:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076929792/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076929792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768227.623652:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768227.623668:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350931c00 x1862023076929920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623688:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076929920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623705:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076929920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768227.623711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930048:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768227.623724:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350930a80 x1862023076930048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076930048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768227.623759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768227.623772:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623777:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350930000 x1862023076930176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623783:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076930176/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623787:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623789:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623792:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768227.623796:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768227.623810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768227.623819:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768227/real 1775768227] req@ffffa04350932680 x1862023076930304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768227.623830:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076930304/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768227.623837:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768227.623840:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768227.623842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076930304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768227.815126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768227.815126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768228.647145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768228.647167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768228.839097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768228.839097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768229.671102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768229.671259:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768229.863155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768229.863156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768230.695142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768230.695152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768230.887144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768230.887163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768231.719184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768231.719357:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768231.911165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768231.911165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768232.743097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768232.743111:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743118:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768232.743122:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768232.743128:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768232.743130:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743139:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743144:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768232.743181:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076930432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743198:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743201:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768232.743202:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743205:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743207:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743210:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743221:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768232.743223:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076930432:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768232.743231:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076930560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743239:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743241:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768232.743242:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743245:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743247:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743249:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743252:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768232.743258:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076930688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743268:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743274:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768232.743275:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743277:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743280:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743283:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743286:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768232.743295:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076930816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743301:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768232.743305:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930560:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768232.743306:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768232.743311:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743316:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743321:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743325:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743330:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768232.743338:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768232.743344:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076930944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743357:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743359:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768232.743361:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743364:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00100000:0.0:1775768232.743368:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930816:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768232.743369:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743376:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743380:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768232.743389:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076931072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743395:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743397:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775768232.743399:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930944:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768232.743400:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743405:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743408:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743411:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743414:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768232.743419:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768232.743423:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076931200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743430:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743432:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768232.743433:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768232.743436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768232.743437:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743442:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743445:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743448:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768232.743459:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076931328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768232.743479:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768232.743482:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768232.743484:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768232.743490:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4734 00000100:00080000:1.0:1775768232.743505:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768232.743509:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768232.743514:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768232.743523:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076931456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768232.743566:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768232.743594:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768232.935133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768232.935133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768233.767162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768233.767185:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768233.959117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768233.959117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768234.791123:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768234.791152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768234.983133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768234.983134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768235.815150:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768235.815215:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:3.0:1775768235.815293:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775768235.815329:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775768235.815345:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815392:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815404:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815412:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815431:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815442:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815450:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815457:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768235.815472:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775768235.815682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350930700 x1862023076931456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076931456/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815726:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815730:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768235.815748:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350931880 x1862023076931328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076931328/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815774:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768235.815780:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350932a00 x1862023076931200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076931200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815797:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815803:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768235.815809:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350932300 x1862023076930944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815820:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076930944/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815828:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815831:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768235.815840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350930e00 x1862023076931072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815850:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076931072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076931072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768235.815868:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350930000 x1862023076930560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076930560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815886:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768235.815904:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350930a80 x1862023076930688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076930688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815920:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768235.815932:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350931c00 x1862023076930816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076930816/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815947:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815950:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076930816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768235.815959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768232/real 1775768235] req@ffffa04350932680 x1862023076930432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768235.815970:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076930432/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768235.815975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768235.815978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768235.815981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076930432:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768236.007132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768236.007138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768236.839204:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768236.841759:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768237.031132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768237.031132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768237.863121:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863128:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775768237.863130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768237.863132:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863141:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863149:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863155:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768237.863181:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076931584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1775768237.863188:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768237.863204:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863207:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768237.863208:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863211:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863213:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863216:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863223:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768237.863231:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076931584:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768237.863232:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076931712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768237.863243:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863245:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768237.863245:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863248:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863251:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863254:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863259:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768237.863269:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076931840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768237.863278:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863281:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768237.863282:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863287:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4739 00000800:00000100:1.0:1775768237.863292:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768237.863295:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863300:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863305:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768237.863316:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076931968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768237.863325:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863329:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775768237.863331:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349641500 x1862023076931584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768237.863332:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863350:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00100000:1.0:1775768237.863353:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076931584/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768237.863357:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863366:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768237.863368:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768237.863371:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775768237.863375:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863380:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076931584:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768237.863385:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076932096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768237.863394:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863397:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768237.863399:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863403:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863406:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768237.863408:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931712:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768237.863413:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863419:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768237.863430:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076932224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768237.863440:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768237.863440:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768237.863457:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768237.863459:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863464:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00100000:1.0:1775768237.863464:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349641c00 x1862023076931712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768237.863470:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768237.863477:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863484:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768237.863487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076931712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863496:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863500:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768237.863510:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076932352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768237.863513:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768237.863527:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768237.863530:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768237.863532:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768237.863536:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863541:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768237.863541:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768237.863548:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863554:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768237.863556:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349641180 x1862023076931840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768237.863573:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076932480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768237.863578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076931840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768237.863594:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768237.863596:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768237.863600:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768237.863603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768237.863604:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768237.863610:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931840:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768237.863615:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4739 00000100:00080000:0.0:1775768237.863620:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768237.863620:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931968:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768237.863630:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768237.863635:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768237.863647:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076932608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768237.863649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349640a80 x1862023076931968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863675:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076931968/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076931968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768237.863701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768237.863718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349643800 x1862023076932096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076932096/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863751:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863756:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768237.863775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932224:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768237.863792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349641880 x1862023076932224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076932224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768237.863832:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768237.863857:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349641500 x1862023076932352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076932352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863883:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863889:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768237.863894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768237.863912:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349640e00 x1862023076932480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076932480/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863939:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768237.863949:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768237.863966:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768237.863975:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768237/real 1775768237] req@ffffa04349642680 x1862023076932608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768237.863985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076932608/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768237.863992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768237.863994:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768237.863997:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076932608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768238.055173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768238.055173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768238.887202:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768238.887272:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768239.079110:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768239.079114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768239.911153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768239.911197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768240.103144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768240.103144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768240.935187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768240.935231:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768241.127089:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768241.127131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768241.959187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768241.959323:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768242.151162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768242.151162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768242.983121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768242.983254:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768242.983384:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983388:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768242.983392:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983408:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983414:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983419:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768242.983446:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076932736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768242.983464:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983466:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768242.983467:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983470:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983473:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983476:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983482:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768242.983491:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076932864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768242.983498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076932736:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768242.983501:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983504:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768242.983505:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983508:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983510:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983518:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983522:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768242.983534:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076932992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768242.983543:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983546:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768242.983548:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983553:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4744 00000800:00000100:1.0:1775768242.983554:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768242.983559:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983564:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983570:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768242.983579:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076933120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768242.983591:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825ea00 x1862023076932736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768242.983591:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983597:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768242.983601:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983607:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983611:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983614:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768242.983617:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076932736/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768242.983619:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775768242.983626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.983630:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.983633:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076932736:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768242.983641:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076933248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768242.983650:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983653:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768242.983655:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983659:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983662:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768242.983668:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768242.983669:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983678:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768242.983689:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076933376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768242.983696:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768242.983700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768242.983701:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768242.983703:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983707:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983710:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768242.983714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825ed80 x1862023076932864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768242.983714:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983721:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768242.983731:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076932864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768242.983733:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076933504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768242.983758:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768242.983765:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768242.983770:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768242.983771:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775768242.983775:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768242.983777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932864:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768242.983783:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00080000:3.0:1775768242.983787:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768242.983789:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768242.983794:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983801:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768242.983814:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076933632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768242.983818:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768242.983826:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768242.983828:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768242.983830:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768242.983834:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4744 00000100:00100000:1.0:1775768242.983835:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825c000 x1862023076932992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768242.983837:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768242.983843:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768242.983850:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768242.983856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076932992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768242.983860:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076933760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768242.983869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.983875:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.983881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076932992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768242.983890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076933120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768242.983909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.983920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825e680 x1862023076933120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.983931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076933120/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.983938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.983941:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.983944:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076933120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768242.983949:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076933248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768242.983962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.983968:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825f100 x1862023076933248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.983974:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076933248/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.983979:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.983981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.983983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076933248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768242.983997:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768242.984014:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.984023:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825e300 x1862023076933376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.984053:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076933376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.984060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.984063:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.984066:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768242.984071:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768242.984087:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.984097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825d500 x1862023076933504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.984108:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076933504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.984115:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.984117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.984120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768242.984126:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768242.984142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.984152:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825dc00 x1862023076933632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.984161:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076933632/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.984168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.984171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.984174:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768242.984178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768242.984194:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768242.984203:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768242/real 1775768242] req@ffffa0435825d880 x1862023076933760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768242.984212:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076933760/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768242.984220:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768242.984223:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768242.984225:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076933760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768243.175149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768243.175149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768244.007156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768244.007473:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768244.199104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768244.199104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768245.031184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768245.031276:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768245.223165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768245.223165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768246.055146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768246.055187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768246.247135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768246.247136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768247.079109:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768247.079251:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768247.271166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768247.271166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768248.103139:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768248.103147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768248.103158:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768248.103164:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103174:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103185:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103193:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768248.103240:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076933888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768248.103270:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103274:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768248.103276:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103281:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103285:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103288:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103298:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768248.103312:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076933888:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768248.103325:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076934016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768248.103356:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103360:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768248.103361:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103366:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103370:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103373:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103378:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768248.103385:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768248.103389:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076934144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768248.103396:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103398:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768248.103400:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103403:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103406:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103408:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103411:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768248.103417:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076934272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768248.103423:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103425:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768248.103427:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103429:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103431:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103434:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103438:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768248.103442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825ce00 x1862023076933888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768248.103447:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076934400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768248.103464:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103469:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768248.103473:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103477:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103480:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103482:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103485:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768248.103488:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076933888/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768248.103493:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076934528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768248.103512:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768248.103513:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768248.103519:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775768248.103519:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768248.103524:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768248.103526:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076933888:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768248.103531:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103536:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103540:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103544:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768248.103551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768248.103558:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076934656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768248.103571:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768248.103576:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768248.103580:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768248.103582:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768248.103585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825d880 x1862023076934016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768248.103589:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103595:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103602:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076934016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768248.103610:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768248.103622:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768248.103627:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076934784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768248.103628:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103635:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934016:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768248.103644:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768248.103648:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768248.103651:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768248.103655:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768248.103660:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4749 00000100:00080000:3.0:1775768248.103665:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768248.103670:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768248.103675:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768248.103676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825dc00 x1862023076934144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768248.103693:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076934912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768248.103723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076934144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103731:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103735:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103739:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768248.103745:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768248.103770:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825d500 x1862023076934272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.103794:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076934272/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103812:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934272:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775768248.103814:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768248.103819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768248.103840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825e300 x1862023076934400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.103862:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076934400/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076934400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768248.103876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768248.103890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825f100 x1862023076934528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.103903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076934528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103907:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103909:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103911:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768248.103922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768248.103931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103935:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825e680 x1862023076934656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.103942:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076934656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768248.103957:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768248.103965:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103969:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825c000 x1862023076934784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.103975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076934784/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.103980:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.103981:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.103983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768248.103986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768248.103993:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768248.103997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768248/real 1775768248] req@ffffa0435825ed80 x1862023076934912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768248.104002:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076934912/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768248.104007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768248.104009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768248.104011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076934912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768248.295114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768248.295114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768249.127167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768249.127329:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768249.319166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768249.319173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768250.151138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768250.151193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768250.343149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768250.343157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768251.175157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768251.175426:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768251.367159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768251.367159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768252.199223:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768252.201405:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768252.391162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768252.391162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768253.223168:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223175:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768253.223179:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223195:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223206:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223213:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768253.223248:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076935040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223272:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223275:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768253.223276:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223278:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223281:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223284:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223291:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768253.223311:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076935168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223317:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223319:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768253.223320:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223322:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223325:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223327:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223331:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768253.223336:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076935296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223341:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223343:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768253.223344:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223346:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223347:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223349:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223351:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768253.223356:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076935424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223360:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223362:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768253.223363:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223365:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223372:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223373:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223378:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768253.223383:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076935552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223388:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223390:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768253.223392:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223394:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223396:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223398:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223400:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768253.223404:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076935680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223409:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223410:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768253.223411:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223413:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223415:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223418:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223421:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768253.223429:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076935808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223436:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223438:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768253.223440:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223443:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223446:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223449:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223453:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768253.223463:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076935936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768253.223470:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768253.223473:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768253.223474:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768253.223477:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4754 00000100:00080000:3.0:1775768253.223479:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768253.223481:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768253.223484:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768253.223490:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076936064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768253.223547:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076935040:192.168.202.158@tcp:250:kworker.0 00010000:00010000:3.0:1775768253.223553:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1775768253.223633:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.223682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825ea00 x1862023076935040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.223702:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076935040/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.223712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.223717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.223721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076935040:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768253.223728:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768253.223748:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.223760:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825ed80 x1862023076935168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.223771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076935168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.223779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.223782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.223785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768253.223791:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768253.223810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.223821:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825c000 x1862023076935296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.223832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076935296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.223839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.223842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.223845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768253.223852:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768253.223871:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.223881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825e680 x1862023076935424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.223891:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076935424/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.223899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.223902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.223904:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768253.223910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768253.223928:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.223938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825f100 x1862023076935552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.223955:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076935552/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.223963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.223965:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.223968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076935552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768253.223975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768253.223993:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.224004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825e300 x1862023076935680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.224016:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076935680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768253.224054:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768253.224133:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.224141:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.224146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768253.224154:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768253.224176:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.224187:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825d500 x1862023076935808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.224198:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076935808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.224211:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.224213:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.224216:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768253.224222:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768253.224240:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.224249:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825dc00 x1862023076935936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.224260:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076935936/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.224268:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.224271:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.224274:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076935936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768253.224280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768253.224311:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768253.224322:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768253/real 1775768253] req@ffffa0435825d880 x1862023076936064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768253.224332:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076936064/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768253.224340:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768253.224343:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768253.224347:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768253.415258:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768253.415376:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768254.247155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768254.247182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768254.439157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768254.439157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768255.271159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768255.271185:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768255.463125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768255.463129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768256.295118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768256.295186:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768256.487125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768256.487125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768257.319120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768257.319154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768257.511128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768257.511133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768258.343122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768258.343126:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343133:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768258.343136:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343144:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343150:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343156:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768258.343186:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076936192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343202:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343206:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768258.343207:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343211:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343213:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343216:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768258.343219:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00100000:0.0:1775768258.343223:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768258.343232:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076936320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343239:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343242:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768258.343255:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768258.343255:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076936192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768258.343263:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343266:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343269:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343273:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768258.343283:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076936448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343291:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343295:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768258.343297:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343301:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343306:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343311:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768258.343316:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343318:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768258.343332:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076936576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343342:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343346:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768258.343346:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349640380 x1862023076936192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768258.343349:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343358:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343362:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343366:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343371:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768258.343382:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076936704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343390:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343393:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768258.343395:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343399:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343402:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343405:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343408:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768258.343413:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076936192/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768258.343417:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076936832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343442:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343444:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768258.343445:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768258.343446:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768258.343450:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:3.0:1775768258.343453:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768258.343454:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768258.343458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076936192:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768258.343459:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343466:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768258.343476:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076936960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768258.343481:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936320:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768258.343492:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343496:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768258.343498:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768258.343502:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768258.343504:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343510:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343513:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343518:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768258.343528:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076937088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768258.343534:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768258.343536:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768258.343537:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768258.343539:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4760 00000100:00080000:0.0:1775768258.343542:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768258.343543:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768258.343546:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768258.343551:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076937216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768258.343606:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349643100 x1862023076936320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343617:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076936320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343626:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343628:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768258.343634:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768258.343693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349640700 x1862023076936448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343709:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076936448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343716:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343718:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768258.343722:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768258.343733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343738:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349643b80 x1862023076936576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076936576/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343752:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768258.343755:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768258.343763:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343768:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349642a00 x1862023076936704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076936704/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343780:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343781:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076936704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768258.343784:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768258.343793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343797:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349642300 x1862023076936832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076936832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343808:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343809:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768258.343823:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768258.343832:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343837:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349642680 x1862023076936960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076936960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343848:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076936960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768258.343853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768258.343861:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349640e00 x1862023076937088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343879:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076937088/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343892:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343895:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768258.343904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768258.343926:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768258.343936:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768258/real 1775768258] req@ffffa04349641500 x1862023076937216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768258.343947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076937216/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768258.343954:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768258.343956:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768258.343959:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768258.535130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768258.535130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768259.367130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768259.367257:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768259.559132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768259.559132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768260.391143:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768260.391150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768260.583097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768260.583097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768261.415151:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768261.415156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768261.607123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768261.607123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768262.439123:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768262.439168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768262.631097:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768262.631173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:2.0:1775768263.463129:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463137:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768263.463141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463150:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463159:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463165:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768263.463166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768263.463179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768263.463217:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076937344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768263.463235:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463237:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768263.463239:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463243:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463246:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463250:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463257:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768263.463262:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076937344:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768263.463268:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076937472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768263.463274:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463276:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768263.463277:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463279:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463286:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463288:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463291:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768263.463299:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076937600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768263.463309:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768263.463312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463314:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463316:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768263.463320:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768263.463321:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463326:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768263.463332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076937728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768263.463338:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768263.463342:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463344:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463345:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463347:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463350:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768263.463357:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076937856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768263.463362:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463364:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775768263.463364:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7df80 x1862023076937344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768263.463367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463372:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463376:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463381:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463384:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768263.463390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076937984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768263.463395:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076937344/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768263.463398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463402:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768263.463404:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463410:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463414:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463416:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768263.463417:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768263.463419:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768263.463426:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768263.463427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076938112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768263.463433:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076937344:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768263.463437:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463441:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768263.463443:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463452:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00080000:2.0:1775768263.463456:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768263.463457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937472:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768263.463463:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768263.463469:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768263.463481:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076938240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768263.463484:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463498:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7d880 x1862023076937472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768263.463502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768263.463511:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768263.463514:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768263.463519:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4765 00000100:00100000:3.0:1775768263.463521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076937472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768263.463524:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768263.463531:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768263.463536:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768263.463537:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768263.463544:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463548:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768263.463549:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076938368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768263.463564:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768263.463585:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463595:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7ed80 x1862023076937600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076937600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463622:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463625:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768263.463630:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768263.463646:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463655:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7ea00 x1862023076937728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076937728/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463676:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463679:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768263.463684:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768263.463700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7f800 x1862023076937856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463721:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076937856/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463729:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463734:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076937856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768263.463740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937984:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768263.463754:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7c700 x1862023076937984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463772:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076937984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076937984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768263.463790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768263.463807:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463814:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7ce00 x1862023076938112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463824:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076938112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463832:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463837:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768263.463853:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768263.463866:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7fb80 x1862023076938240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076938240/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768263.463898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768263.463909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768263.463914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768263/real 1775768263] req@ffffa0434aa7df80 x1862023076938368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768263.463921:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076938368/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768263.463925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768263.463927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768263.463928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076938368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768263.655186:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768263.655186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768264.487102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768264.487176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768264.679122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768264.679122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768265.511168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768265.511168:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768265.703117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768265.703117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768266.535094:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768266.535127:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768266.727140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768266.727140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768267.559126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768267.559154:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768267.751093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768267.751102:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768268.583129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768268.583139:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583145:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775768268.583166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768268.583175:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583184:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583193:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583199:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768268.583234:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076938496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768268.583261:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583264:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768268.583266:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583270:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583274:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583278:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583286:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768268.583296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076938496:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768268.583300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076938624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768268.583310:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583313:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768268.583315:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583319:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583323:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583326:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583331:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768268.583342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076938752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768268.583351:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768268.583354:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768268.583359:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583368:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583372:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583378:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768268.583389:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076938880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768268.583398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768268.583399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349641880 x1862023076938496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768268.583403:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768268.583407:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583419:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583422:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768268.583438:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076938496/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768268.583440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076939008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768268.583457:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583462:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583466:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076938496:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768268.583486:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768268.583490:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583493:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768268.583496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583500:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583505:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768268.583507:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768268.583512:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583518:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768268.583519:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349643800 x1862023076938624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583529:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076938624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768268.583534:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076939136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768268.583537:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768268.583553:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768268.583555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768268.583560:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768268.583563:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583568:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4770 00000800:00000100:3.0:1775768268.583572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768268.583573:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583579:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768268.583581:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349640a80 x1862023076938752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768268.583586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768268.583599:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076939264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768268.583601:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076938752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583625:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583629:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768268.583630:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768268.583634:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768268.583636:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768268.583639:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768268.583640:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938880:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768268.583649:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00080000:0.0:1775768268.583654:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768268.583660:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768268.583661:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583667:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768268.583669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349641180 x1862023076938880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076938880/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768268.583683:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076939392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768268.583684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583696:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076938880:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768268.583705:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768268.583708:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768268.583710:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768268.583714:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4770 00000100:00100000:3.0:1775768268.583716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939008:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768268.583720:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768268.583725:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768268.583731:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768268.583744:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768268.583745:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076939520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768268.583756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349641c00 x1862023076939008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583767:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076939008/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583776:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583779:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583782:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768268.583787:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768268.583813:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768268.583823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349641880 x1862023076939136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583832:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076939136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768268.583849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768268.583865:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768268.583873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349641500 x1862023076939264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076939264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583889:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583894:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768268.583898:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768268.583914:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768268.583922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349640e00 x1862023076939392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076939392/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.583942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768268.583956:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768268.583970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768268.583978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768268/real 1775768268] req@ffffa04349642680 x1862023076939520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768268.583987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076939520/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768268.583995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768268.583997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768268.584000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076939520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768268.775167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768268.775173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768269.607177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768269.607297:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768269.799188:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768269.799188:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768270.631139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768270.631183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768270.823162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768270.823162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768271.655132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768271.655172:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768271.847186:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768271.847186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768272.679171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768272.679294:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768272.871179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768272.871195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768273.703142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768273.703161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768273.703172:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703182:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768273.703188:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703201:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703216:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703224:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768273.703268:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076939648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703296:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703300:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768273.703301:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703304:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703307:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703310:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703319:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768273.703329:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076939776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703336:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703338:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768273.703339:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703342:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703343:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703345:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703347:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768273.703353:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076939904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703358:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703361:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768273.703362:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703364:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703366:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703367:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703371:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768273.703376:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076940032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703382:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703384:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768273.703385:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703387:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703389:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703392:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703394:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768273.703402:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076940160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703412:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768273.703415:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076939648:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768273.703416:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768273.703420:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703426:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703429:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703434:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703439:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768273.703452:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076940288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703459:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703462:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768273.703464:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703466:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703469:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703472:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703476:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768273.703484:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076940416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703490:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703494:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768273.703495:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768273.703495:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768273.703501:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703503:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703505:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703507:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768273.703513:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076940544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768273.703519:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768273.703522:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768273.703523:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768273.703525:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4775 00000100:00080000:0.0:1775768273.703527:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768273.703529:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768273.703532:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768273.703541:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076940672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768273.703545:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349642300 x1862023076939648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703562:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076939648/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703574:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703577:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076939648:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768273.703583:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768273.703603:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703621:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349642a00 x1862023076939776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076939776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703646:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703649:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768273.703653:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768273.703664:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703669:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349643b80 x1862023076939904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076939904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703681:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703682:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703684:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076939904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768273.703687:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768273.703696:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703701:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349640700 x1862023076940032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076940032/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703712:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703714:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768273.703739:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768273.703748:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703754:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349643100 x1862023076940160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076940160/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703765:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703767:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703768:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768273.703771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940288:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768273.703780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703785:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349640380 x1862023076940288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076940288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703797:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768273.703801:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768273.703809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703813:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349642680 x1862023076940416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703819:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076940416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703824:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768273.703831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768273.703839:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349640e00 x1862023076940544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703849:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076940544/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703853:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703855:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768273.703860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768273.703868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768273.703872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768273/real 1775768273] req@ffffa04349641500 x1862023076940672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768273.703878:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076940672/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768273.703882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768273.703884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768273.703886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076940672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768273.895144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768273.895170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768274.727169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768274.727215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768274.919141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768274.919141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768275.751170:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768275.751193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768275.943174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768275.943175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768276.775190:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768276.775217:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768276.967151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768276.967162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768277.799148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768277.799176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768277.991108:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768277.991111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768278.823165:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768278.823171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768278.823180:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768278.823185:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823196:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4780 00010000:00010000:3.0:1775768278.823201:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768278.823208:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823217:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768278.823258:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076940800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823281:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823284:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768278.823285:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823289:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823291:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823294:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823303:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768278.823311:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076940928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768278.823317:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076940800:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768278.823320:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823325:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768278.823327:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823330:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823332:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823335:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823338:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768278.823348:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076941056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823360:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823363:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768278.823364:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823369:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823371:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823374:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823378:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768278.823386:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076941184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823392:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768278.823393:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768278.823397:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768278.823398:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823401:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823404:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823406:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823409:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768278.823416:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076941312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823421:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823424:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768278.823425:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823428:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823431:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823434:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823437:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349641880 x1862023076940800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768278.823438:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768278.823449:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076941440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823455:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768278.823457:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076940800/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768278.823460:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768278.823463:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823471:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823473:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768278.823474:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768278.823476:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823480:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768278.823481:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823487:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076940800:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768278.823492:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076941568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768278.823497:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823499:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768278.823500:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823503:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00080000:0.0:1775768278.823506:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823510:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768278.823516:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768278.823519:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768278.823536:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076941696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768278.823539:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823550:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349641c00 x1862023076940928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768278.823551:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768278.823560:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768278.823562:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768278.823567:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4780 00000100:00100000:3.0:1775768278.823569:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076940928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768278.823571:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768278.823578:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768278.823581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768278.823583:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768278.823586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823590:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076940928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768278.823596:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076941824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768278.823597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768278.823617:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823625:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349641180 x1862023076941056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823632:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076941056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823638:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768278.823643:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768278.823653:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823657:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349640a80 x1862023076941184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076941184/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823668:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823669:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768278.823675:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768278.823684:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823688:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349643800 x1862023076941312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823700:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076941312/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823706:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076941312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768278.823712:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768278.823722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823728:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349641500 x1862023076941440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823734:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076941440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823739:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823740:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823741:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768278.823744:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768278.823752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349640e00 x1862023076941568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823764:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076941568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768278.823785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768278.823794:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349642680 x1862023076941696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076941696/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823810:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823812:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768278.823816:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768278.823824:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768278.823829:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768278/real 1775768278] req@ffffa04349640380 x1862023076941824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768278.823835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076941824/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768278.823840:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768278.823842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768278.823844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076941824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768279.015162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768279.015162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768279.847152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768279.847207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768280.039133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768280.039133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768280.871114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768280.871154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768281.063100:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768281.063106:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768281.895134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768281.895201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768282.087187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768282.087187:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768282.919168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768282.919244:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768283.111152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768283.111153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:3.0:1775768283.943130:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768283.943131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768283.943141:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768283.943146:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943157:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943170:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943178:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768283.943227:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076941952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943254:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943256:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768283.943258:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943262:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943266:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943271:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943282:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768283.943290:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076941952:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768283.943300:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076942080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943312:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943326:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768283.943328:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943333:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943337:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943341:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943345:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768283.943355:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076942208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943361:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943365:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775768283.943367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768283.943367:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943374:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943376:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943378:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943382:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768283.943389:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076942336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943395:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943397:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768283.943398:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943402:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943404:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943407:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943410:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768283.943418:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076942464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768283.943422:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825ce00 x1862023076941952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768283.943427:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943433:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768283.943436:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943442:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943446:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943452:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943453:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076941952/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768283.943464:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768283.943468:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.943474:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768283.943478:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076942592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768283.943482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076941952:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768283.943496:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943499:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768283.943501:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768283.943507:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943510:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943514:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943518:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942080:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768283.943521:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768283.943535:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076942720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943543:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943545:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768283.943548:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768283.943549:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768283.943553:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943556:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943560:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943563:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768283.943568:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825d880 x1862023076942080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768283.943573:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076942848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768283.943596:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768283.943599:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768283.943600:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768283.943604:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076942080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768283.943605:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4785 00000100:00080000:3.0:1775768283.943609:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768283.943614:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768283.943620:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768283.943625:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768283.943628:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076942976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768283.943633:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943641:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768283.943652:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768283.943677:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825dc00 x1862023076942208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.943704:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076942208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.943713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:2.0:1775768283.943715:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768283.943717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768283.943728:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768283.943749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943759:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825d500 x1862023076942336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.943771:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076942336/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.943778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.943782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768283.943790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768283.943809:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825e300 x1862023076942464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.943830:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076942464/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.943838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.943842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076942464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768283.943849:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768283.943868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825f100 x1862023076942592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.943887:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076942592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.943895:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.943898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943901:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768283.943917:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768283.943934:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.943944:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825e680 x1862023076942720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.943954:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076942720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.943967:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.943970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.943973:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768283.943978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768283.943995:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.944017:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825c000 x1862023076942848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.944045:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076942848/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.944051:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.944054:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.944057:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768283.944063:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768283.944082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768283.944092:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768283/real 1775768283] req@ffffa0435825ed80 x1862023076942976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768283.944103:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076942976/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768283.944111:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768283.944114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768283.944117:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076942976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768284.135153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768284.135153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768284.967159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768284.967167:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768285.159099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768285.159148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768285.991093:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768285.991209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768286.183130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768286.183130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768287.015132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768287.015231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768287.207124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768287.207124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768288.039134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768288.039220:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768288.231173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768288.231173:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768289.063135:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768289.063141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768289.063150:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768289.063156:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063169:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063181:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063190:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768289.063237:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076943104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063267:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063271:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768289.063272:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063277:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063281:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063285:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063297:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768289.063304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076943104:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768289.063314:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076943232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063324:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063327:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768289.063328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063332:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063336:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063340:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063345:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768289.063357:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076943360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063367:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063370:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768289.063372:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063375:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4790 00000800:00000100:1.0:1775768289.063380:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768289.063381:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063388:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063393:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768289.063405:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076943488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063413:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063416:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768289.063417:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063421:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063424:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063428:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063432:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768289.063439:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825ea00 x1862023076943104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768289.063445:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076943616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063461:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768289.063466:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768289.063468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076943104/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768289.063474:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063481:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063486:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768289.063487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775768289.063492:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768289.063494:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063500:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076943104:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768289.063506:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076943744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063516:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063519:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768289.063521:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063524:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063528:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063532:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063537:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768289.063538:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768289.063552:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076943872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063561:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768289.063564:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768289.063566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063569:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063573:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768289.063577:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768289.063580:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768289.063586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768289.063595:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825ed80 x1862023076943232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768289.063599:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076944000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768289.063620:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768289.063622:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076943232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768289.063624:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768289.063627:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768289.063633:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4790 00000100:00080000:3.0:1775768289.063636:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768289.063644:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768289.063649:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775768289.063650:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768289.063656:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063660:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768289.063666:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076944128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768289.063674:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768289.063699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.063711:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825c000 x1862023076943360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.063723:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076943360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.063732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.063736:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768289.063748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943488:192.168.202.158@tcp:8:kworker.0 00010000:00010000:0.0:1775768289.063759:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1775768289.063771:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.063781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825e680 x1862023076943488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.063793:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076943488/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.063800:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.063803:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063806:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768289.063812:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768289.063831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.063840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825f100 x1862023076943616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.063860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076943616/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.063877:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.063881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076943616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768289.063890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076943744:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768289.063909:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.063918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825e300 x1862023076943744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.063928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076943744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.063936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.063939:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.063942:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076943744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768289.063982:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076943872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768289.064000:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.064009:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825d500 x1862023076943872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.064019:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076943872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.064052:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.064055:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.064059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076943872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768289.064064:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768289.064082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.064093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825dc00 x1862023076944000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.064103:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076944000/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.064111:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.064115:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.064117:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768289.064123:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768289.064140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768289.064149:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768289/real 1775768289] req@ffffa0435825d880 x1862023076944128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768289.064161:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076944128/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768289.064169:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768289.064172:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768289.064175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768289.255120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768289.255156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768290.087164:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768290.087226:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768290.279099:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768290.279099:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768291.111129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768291.111272:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768291.303123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768291.303123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768292.135173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768292.135215:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768292.327127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768292.327127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768293.159135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768293.159186:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768293.351139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768293.351166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:3.0:1775768294.183186:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768294.183187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768294.183200:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768294.183206:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183217:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183229:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183235:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768294.183267:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076944256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768294.183292:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183295:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768294.183298:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183301:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183303:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183307:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768294.183321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076944256:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775768294.183322:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768294.183338:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076944384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768294.183346:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183348:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768294.183349:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183352:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183354:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183356:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768294.183359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768294.183366:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076944512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768294.183371:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183374:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768294.183375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183378:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183380:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768294.183384:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768294.183385:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768294.183391:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768294.183402:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076944640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768294.183408:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183410:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768294.183413:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183415:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183418:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183420:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768294.183423:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768294.183433:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076944768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768294.183435:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825ce00 x1862023076944256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768294.183447:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183451:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768294.183453:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183459:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183466:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183471:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768294.183474:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076944256/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768294.183478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768294.183487:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183493:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768294.183498:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076944896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768294.183503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076944256:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768294.183529:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768294.183564:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768294.183584:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768294.183589:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768294.183592:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768294.183603:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825d880 x1862023076944384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768294.183625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076944384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.183632:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.183638:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768294.183645:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768294.183669:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768294.183693:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825dc00 x1862023076944512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768294.183702:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076944512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.183707:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183709:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.183712:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768294.183717:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944640:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768294.183723:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4795 00000800:00000100:1.0:1775768294.183734:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768294.183737:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183746:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768294.183749:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825d500 x1862023076944640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768294.183752:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768294.183769:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076944640/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.183777:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183781:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.183786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768294.183792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768294.183818:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768294.183835:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076945024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768294.183842:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768294.183846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825e300 x1862023076944768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768294.183846:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768294.183849:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183856:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00100000:1.0:1775768294.183861:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076944768/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768294.183862:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768294.183870:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768294.183871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:3.0:1775768294.183878:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768294.183879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076944768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768294.183894:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076945152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768294.183914:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944896:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768294.183927:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768294.183931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768294.183934:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768294.183937:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768294.183942:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4795 00000100:00080000:3.0:1775768294.183947:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768294.183948:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825f100 x1862023076944896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768294.183953:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768294.183962:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076944896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768294.183963:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768294.183973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.183975:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.183978:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076944896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768294.183982:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076945280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768294.183986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768294.184005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768294.184013:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825e680 x1862023076945024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768294.184021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076945024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.184045:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.184052:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.184059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945024:192.168.202.158@tcp:38:kworker.0 00010000:00010000:0.0:1775768294.184081:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768294.184082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768294.184103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768294.184121:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825c000 x1862023076945152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768294.184132:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076945152/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.184139:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.184142:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.184145:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768294.184150:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768294.184168:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768294.184177:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768294/real 1775768294] req@ffffa0435825ed80 x1862023076945280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768294.184188:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076945280/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768294.184196:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768294.184198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768294.184199:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076945280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768294.375161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768294.375165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768295.207160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768295.207220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768295.399121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768295.399121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768296.231160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768296.231230:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768296.423148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768296.423148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768297.255147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768297.255197:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768297.447112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768297.447130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768298.279148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768298.279152:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768298.471118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768298.471118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768299.303121:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303132:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768299.303138:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303150:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00100000:1.0:1775768299.303157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768299.303161:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303170:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775768299.303183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768299.303214:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076945408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303239:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768299.303244:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303249:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303253:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303257:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768299.303265:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076945408:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775768299.303268:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768299.303282:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076945536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303297:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768299.303299:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303304:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303320:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303323:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768299.303328:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768299.303339:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076945664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303348:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303352:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768299.303354:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303362:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303367:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768299.303372:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945536:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768299.303373:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768299.303390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076945792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303402:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768299.303402:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945664:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768299.303409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768299.303411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303416:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303421:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303425:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768299.303427:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945792:192.168.202.158@tcp:8:kworker.0 00000008:00100000:0.0:1775768299.303434:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768299.303447:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076945920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303463:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768299.303465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768299.303471:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945920:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768299.303472:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303478:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303483:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768299.303489:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768299.303502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076946048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303517:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303521:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768299.303523:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768299.303527:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768299.303529:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303535:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303539:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768299.303544:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768299.303556:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076946176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303570:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303574:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768299.303576:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768299.303580:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768299.303583:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303587:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768299.303591:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768299.303596:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768299.303609:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076946304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768299.303622:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768299.303626:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768299.303628:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768299.303633:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4800 00000100:00080000:0.0:1775768299.303637:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768299.303638:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946304:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768299.303643:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768299.303650:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768299.303663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076946432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768299.303691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768299.495183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768299.495183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768300.327138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768300.327314:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768300.519151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768300.519165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768301.351117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768301.351275:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768301.543123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768301.543123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768302.375126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768302.375287:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1775768302.375312:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:1.0:1775768302.375340:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775768302.375352:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375393:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375404:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375411:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375419:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375427:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375435:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375440:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:1.0:1775768302.375448:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775768302.375471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349641500 x1862023076946432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375495:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076946432/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375504:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375509:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375513:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768302.375522:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349640e00 x1862023076946304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076946304/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375548:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768302.375555:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349642680 x1862023076946176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375569:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076946176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375576:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375583:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768302.375590:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349640380 x1862023076946048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076946048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375610:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076946048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768302.375620:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349642300 x1862023076945920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375630:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076945920/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375638:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375642:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375644:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768302.375649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349642a00 x1862023076945792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375660:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076945792/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375666:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375668:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768302.375676:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349640700 x1862023076945536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076945536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375697:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375700:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768302.375706:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349643b80 x1862023076945664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375717:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076945664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375732:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076945664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768302.375738:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768299/real 1775768302] req@ffffa04349643100 x1862023076945408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768302.375750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076945408/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768302.375756:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768302.375758:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768302.375761:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076945408:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768302.567141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768302.567422:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768303.399100:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768303.399229:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768303.591122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768303.591122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768304.423187:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423194:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768304.423197:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423204:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423211:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423216:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768304.423251:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076946560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423279:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423283:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768304.423284:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423288:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423291:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423295:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423303:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768304.423314:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076946688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423323:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423326:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768304.423328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423331:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423334:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423338:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768304.423354:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076946816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768304.423364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076946560:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768304.423371:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768304.423374:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423378:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423382:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423386:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423391:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768304.423402:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076946944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423410:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423413:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768304.423415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423418:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4806 00000800:00000100:3.0:1775768304.423439:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768304.423467:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768304.423493:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768304.423844:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423851:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423857:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768304.423870:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076947072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423880:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423883:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768304.423885:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423888:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423892:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423897:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423902:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768304.423913:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076947200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423921:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423924:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768304.423925:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.423928:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423932:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.423935:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423940:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768304.423941:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350932680 x1862023076946560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768304.423953:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076947328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768304.423961:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.423964:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768304.423967:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768304.423967:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076946560/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768304.423975:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.423982:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768304.423986:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768304.423987:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.423995:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768304.423996:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424003:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076946560:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768304.424008:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076947456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768304.424015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946688:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768304.424023:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768304.424052:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768304.424054:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768304.424058:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4806 00000100:00080000:1.0:1775768304.424061:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768304.424065:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768304.424070:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768304.424081:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076947584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768304.424089:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424108:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350931c00 x1862023076946688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424121:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076946688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424129:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424133:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424137:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768304.424164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768304.424184:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424194:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350930a80 x1862023076946816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424206:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076946816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424213:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424216:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424219:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768304.424224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768304.424241:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424250:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350930000 x1862023076946944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424260:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076946944/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424267:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424270:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424272:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076946944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768304.424278:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768304.424296:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424305:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350930e00 x1862023076947072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076947072/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424322:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424325:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424327:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768304.424331:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947200:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768304.424349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424359:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350932300 x1862023076947200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424368:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076947200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424376:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424378:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424396:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768304.424401:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768304.424419:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424429:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350932a00 x1862023076947328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424445:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076947328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424452:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424454:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424457:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768304.424462:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768304.424480:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424489:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350931880 x1862023076947456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424499:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076947456/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424506:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424509:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424511:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768304.424515:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768304.424532:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768304.424541:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768304/real 1775768304] req@ffffa04350930700 x1862023076947584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768304.424551:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076947584/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768304.424564:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768304.424567:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768304.424569:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076947584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768304.615115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768304.615115:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768305.447132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768305.447261:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768305.639138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768305.639138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768306.471118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768306.471191:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768306.663101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768306.663101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768307.495118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768307.495154:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768307.687119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768307.687119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768308.519126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768308.519220:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768308.711112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768308.711112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768309.543121:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768309.543165:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543173:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768309.543176:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543185:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543204:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543212:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768309.543247:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076947712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768309.543271:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543275:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768309.543277:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543282:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543286:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543291:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543297:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076947712:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768309.543299:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768309.543315:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076947840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768309.543325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543329:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768309.543331:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543334:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543338:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768309.543349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768309.543353:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768309.543359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768309.543371:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076947968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768309.543381:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543385:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768309.543387:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768309.543391:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7e680 x1862023076947712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543394:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543400:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543407:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768309.543412:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768309.543417:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076947712/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768309.543426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076948096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768309.543428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768309.543441:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768309.543442:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076947712:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768309.543449:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768309.543450:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543455:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543458:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543463:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543464:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947840:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768309.543471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768309.543485:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076948224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768309.543488:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768309.543500:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768309.543500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7c380 x1862023076947840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543508:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768309.543511:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543518:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00100000:3.0:1775768309.543519:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076947840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543524:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543529:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768309.543533:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768309.543535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543543:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768309.543547:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076948352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768309.543550:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768309.543562:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543565:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768309.543567:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768309.543570:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768309.543575:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543579:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768309.543579:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7f480 x1862023076947968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543587:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076947968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768309.543592:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768309.543601:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543604:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768309.543607:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076948480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768309.543608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076947968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768309.543616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948096:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768309.543627:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543631:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775768309.543633:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7f100 x1862023076948096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543643:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768309.543648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076948096/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768309.543652:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:3.0:1775768309.543659:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768309.543660:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768309.543664:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768309.543669:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543669:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948096:192.168.202.158@tcp:8:kworker.0 00000008:00100000:2.0:1775768309.543676:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768309.543687:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076948608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768309.543690:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948224:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768309.543701:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768309.543703:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768309.543705:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768309.543708:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4811 00000100:00080000:2.0:1775768309.543713:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768309.543718:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768309.543720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768309.543722:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768309.543730:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076948736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768309.543736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7c000 x1862023076948224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768309.543744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076948224/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768309.543764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543769:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768309.543773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768309.543786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543794:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7e300 x1862023076948352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768309.543801:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768309.543803:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076948352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768309.543814:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543817:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768309.543822:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768309.543831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7d500 x1862023076948480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768309.543842:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076948480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768309.543847:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543849:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543850:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768309.543858:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768309.543866:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543873:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7dc00 x1862023076948608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768309.543896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076948608/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768309.543904:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768309.543915:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768309.543931:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768309.543938:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768309/real 1775768309] req@ffffa0434aa7f100 x1862023076948736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768309.543945:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076948736/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768309.543950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768309.543952:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768309.543954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076948736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768309.735122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768309.735122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768310.567111:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768310.567147:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768310.759121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768310.759121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768311.591118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768311.591155:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768311.783118:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768311.783118:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768312.615088:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768312.615131:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768312.807124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768312.807124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768313.639130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768313.639178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768313.831138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768313.831138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768314.663129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768314.663154:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775768314.663159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768314.663160:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768314.663166:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663183:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663192:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663198:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768314.663232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076948864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768314.663255:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663258:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768314.663260:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663264:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663268:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663272:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663280:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768314.663291:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076948864:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768314.663293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076948992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768314.663305:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768314.663310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663314:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663318:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663321:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663326:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768314.663337:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076949120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768314.663345:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663348:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768314.663350:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768314.663355:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768314.663358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663362:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663367:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663372:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768314.663382:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076949248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768314.663390:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663394:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768314.663397:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768314.663398:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349643800 x1862023076948864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768314.663405:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663412:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663418:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663423:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768314.663425:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076948864/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768314.663435:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076949376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768314.663437:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663446:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768314.663452:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768314.663454:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076948864:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768314.663456:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768314.663460:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663466:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663470:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663474:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768314.663483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768314.663492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076949504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768314.663502:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663505:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768314.663508:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768314.663511:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768314.663515:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663519:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768314.663523:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663528:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768314.663529:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349640a80 x1862023076948992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768314.663542:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076949632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768314.663548:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076948992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.663573:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768314.663577:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768314.663580:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768314.663583:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768314.663586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076948992:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768314.663587:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663595:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663602:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768314.663605:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949120:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768314.663607:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663616:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768314.663629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076949760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768314.663637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768314.663638:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768314.663642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768314.663644:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768314.663649:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4816 00000100:00080000:0.0:1775768314.663653:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768314.663653:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349641180 x1862023076949120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768314.663660:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768314.663668:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768314.663676:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076949120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768314.663681:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076949888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768314.663693:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663713:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.663719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768314.663726:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768314.663748:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.663760:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349641c00 x1862023076949248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.663772:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076949248/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.663780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663783:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.663786:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768314.663792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768314.663808:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.663817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349641880 x1862023076949376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.663827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076949376/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.663835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.663840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076949376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768314.663865:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768314.663883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.663892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349643800 x1862023076949504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.663903:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076949504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.663911:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663914:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.663917:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768314.663921:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768314.663938:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.663947:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349643100 x1862023076949632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.663957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076949632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.663970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.663973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.663975:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768314.663979:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768314.663992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.663998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349643b80 x1862023076949760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.664005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076949760/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.664009:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.664011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.664013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768314.664015:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768314.664023:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768314.664048:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768314/real 1775768314] req@ffffa04349640700 x1862023076949888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768314.664054:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076949888/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768314.664059:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768314.664062:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768314.664064:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076949888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768314.855123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768314.855123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768315.687179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768315.687450:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768315.879121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768315.879121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768316.711122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768316.711142:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768316.903116:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768316.903131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768317.735203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768317.735270:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768317.927142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768317.927200:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768318.759183:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768318.759211:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768318.951205:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768318.951205:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768319.783158:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768319.783179:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783186:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768319.783190:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783200:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783210:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783216:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768319.783260:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076950016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768319.783288:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783291:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768319.783299:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783303:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783306:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1775768319.783309:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768319.783309:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783320:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768319.783323:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076950016:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768319.783332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076950144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768319.783339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768319.783343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783346:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783348:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783350:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783352:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768319.783360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076950272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768319.783366:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783368:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768319.783369:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783371:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783374:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783376:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783379:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768319.783387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076950400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768319.783391:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768319.783397:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783399:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768319.783400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783403:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783406:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783408:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783411:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768319.783417:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076950528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768319.783422:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783424:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768319.783425:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783428:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783430:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783432:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783435:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768319.783441:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076950656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768319.783444:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7f480 x1862023076950016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768319.783452:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783458:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768319.783459:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783464:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783467:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768319.783468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076950016/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768319.783471:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783478:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768319.783482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768319.783486:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076950784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768319.783489:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783496:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076950016:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768319.783498:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783501:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768319.783502:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783505:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4821 00000100:00080000:2.0:1775768319.783507:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783509:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783512:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768319.783517:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076950912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768319.783520:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950144:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768319.783530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768319.783532:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768319.783533:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768319.783537:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4821 00000800:00000100:1.0:1775768319.783540:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768319.783540:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768319.783545:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768319.783548:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768319.783552:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7c380 x1862023076950144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768319.783558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076951040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768319.783565:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076950144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783574:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783577:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768319.783583:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768319.783594:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783599:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7e680 x1862023076950272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783606:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076950272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768319.783622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768319.783631:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783636:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7df80 x1862023076950400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783642:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076950400/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783647:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783648:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768319.783673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768319.783688:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7fb80 x1862023076950528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076950528/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783716:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783719:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783723:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076950528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768319.783728:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768319.783747:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783757:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7ce00 x1862023076950656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783766:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076950656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768319.783778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768319.783790:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7c700 x1862023076950784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783813:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076950784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783817:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783819:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783821:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768319.783833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768319.783850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783860:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7f800 x1862023076950912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783871:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076950912/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783879:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783882:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783885:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076950912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768319.783890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768319.783908:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768319.783918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768319/real 1775768319] req@ffffa0434aa7ea00 x1862023076951040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768319.783930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076951040/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768319.783937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768319.783940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768319.783943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768319.975163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768319.975163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768320.807128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768320.807178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768320.999152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768320.999158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768321.831165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768321.831336:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768322.023114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768322.023114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768322.855146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768322.855201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768323.047171:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768323.047171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768323.879142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768323.879218:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768324.071112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768324.071112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768324.903110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768324.903122:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903130:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768324.903134:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903145:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4826 00010000:00010000:3.0:1775768324.903158:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768324.903160:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903169:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768324.903232:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076951168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768324.903261:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903266:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768324.903268:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903273:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:2.0:1775768324.903278:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903283:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768324.903289:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076951168:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768324.903295:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768324.903312:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076951296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768324.903323:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903327:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768324.903330:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903334:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:2.0:1775768324.903338:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903343:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903348:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768324.903355:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768324.903362:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076951424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768324.903373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768324.903379:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903383:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:2.0:1775768324.903387:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768324.903395:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7ed80 x1862023076951168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768324.903401:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903411:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768324.903417:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076951168/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903426:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768324.903426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076951552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768324.903432:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903440:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076951168:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768324.903447:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903451:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768324.903453:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903458:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:2.0:1775768324.903462:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768324.903465:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951296:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768324.903470:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903476:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1775768324.903483:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768324.903492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076951680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768324.903493:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7d880 x1862023076951296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768324.903501:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076951296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903518:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.903523:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951296:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768324.903527:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768324.903533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951424:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768324.903534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768324.903538:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903544:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4826 00000800:00000100:1.0:1775768324.903550:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768324.903550:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903557:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768324.903559:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7ea00 x1862023076951424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768324.903565:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768324.903570:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076951424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903580:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.903584:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903586:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768324.903589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076951808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768324.903590:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951552:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768324.903613:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903615:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768324.903617:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903621:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:2.0:1775768324.903625:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903627:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903632:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768324.903634:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768324.903645:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076951936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768324.903646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7f800 x1862023076951552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768324.903673:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768324.903674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076951552/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768324.903681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768324.903685:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768324.903692:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768324.903693:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4826 00000100:00080000:1.0:1775768324.903701:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768324.903701:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768324.903708:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951552:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768324.903710:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903720:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768324.903732:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768324.903735:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076952064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768324.903748:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768324.903752:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768324.903754:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768324.903760:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4826 00000800:00000100:1.0:1775768324.903761:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768324.903764:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768324.903768:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768324.903772:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768324.903778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7c700 x1862023076951680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768324.903783:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076952192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768324.903812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076951680/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903824:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.903827:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903831:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076951680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768324.903837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768324.903856:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768324.903867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7ce00 x1862023076951808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768324.903877:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076951808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903885:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.903888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768324.903897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768324.903916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768324.903925:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7fb80 x1862023076951936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768324.903937:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076951936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.903945:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.903948:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.903950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076951936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768324.903964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768324.903983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768324.903995:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7df80 x1862023076952064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768324.904006:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076952064/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.904014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.904017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.904020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768324.904046:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768324.904066:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768324.904076:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768324/real 1775768324] req@ffffa0434aa7e680 x1862023076952192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768324.904083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076952192/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768324.904088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768324.904091:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768324.904092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768325.095175:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768325.095175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768325.927176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768325.927235:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768326.119153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768326.119153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768326.951159:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768326.951189:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768327.143149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768327.143149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768327.975137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768327.975234:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768328.167138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768328.167138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768328.999132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768328.999176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768329.191129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768329.191129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:2.0:1775768330.023115:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023126:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768330.023130:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023142:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023153:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023160:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768330.023198:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076952320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023222:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023225:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768330.023227:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023231:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023234:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023238:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023246:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768330.023257:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076952448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023266:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023269:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768330.023270:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023274:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023277:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023280:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023284:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768330.023293:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076952576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023306:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768330.023310:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023313:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023316:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023319:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023323:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768330.023332:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076952704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023339:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023343:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768330.023344:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023347:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023350:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023365:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023373:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768330.023381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076952832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023389:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023392:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768330.023393:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023397:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023400:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023404:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023408:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768330.023416:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076952960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023423:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023426:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768330.023428:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023431:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023434:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023437:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023440:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768330.023448:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076953088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023456:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023459:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768330.023460:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023463:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023465:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023468:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768330.023478:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076953216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768330.023492:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768330.023495:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768330.023496:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768330.023499:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4831 00000100:00080000:2.0:1775768330.023502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768330.023504:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768330.023508:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768330.023515:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076953344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1775768330.023635:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768330.023737:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076952320:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768330.023751:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000800:00000100:1.0:1775768330.023795:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.023845:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7c380 x1862023076952320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.023864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076952320/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.023873:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.023877:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.023880:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076952320:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768330.023887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768330.023907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.023918:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7f480 x1862023076952448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.023930:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076952448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.023948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.023951:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.023954:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768330.023960:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768330.023979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.023988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7f100 x1862023076952576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024000:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076952576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768330.024050:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768330.024069:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024089:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7dc00 x1862023076952704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024102:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076952704/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024110:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024113:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024115:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768330.024121:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768330.024138:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7d500 x1862023076952832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024158:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076952832/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024166:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076952832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768330.024177:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768330.024196:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024206:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7e300 x1862023076952960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024217:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076952960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024227:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024230:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076952960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768330.024234:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768330.024250:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024258:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7c000 x1862023076953088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076953088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024279:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768330.024284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768330.024297:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024305:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7e680 x1862023076953216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076953216/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024322:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024325:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024327:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768330.024332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768330.024349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768330.024358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768330/real 1775768330] req@ffffa0434aa7df80 x1862023076953344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768330.024369:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076953344/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768330.024375:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768330.024378:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768330.024381:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076953344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768330.215150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768330.215150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768331.047139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768331.047199:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768331.239122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768331.239122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768332.071205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768332.071243:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768332.263108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768332.263138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768333.095172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768333.095233:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768333.287183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768333.287188:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768334.119154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768334.119224:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768334.311165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768334.311166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768335.143131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768335.143149:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143155:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768335.143159:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768335.143167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768335.143170:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143186:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143193:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768335.143229:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076953472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768335.143251:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143255:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768335.143256:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768335.143260:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143264:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143268:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768335.143276:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768335.143284:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076953472:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768335.143290:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076953600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768335.143300:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143303:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768335.143305:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768335.143309:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143312:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143316:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768335.143320:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768335.143330:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076953728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768335.143335:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768335.143345:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143348:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768335.143350:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768335.143355:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143358:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143362:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143371:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349642a00 x1862023076953472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768335.143373:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768335.143385:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076953472/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768335.143390:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076953856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768335.143399:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768335.143403:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143408:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768335.143408:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768335.143411:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768335.143416:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076953472:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768335.143419:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143425:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143429:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768335.143432:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768335.143442:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076953984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768335.143445:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953600:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768335.143459:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143462:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768335.143464:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768335.143468:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143472:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143477:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768335.143477:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768335.143484:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768335.143496:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349642300 x1862023076953600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768335.143497:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076954112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768335.143519:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143527:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768335.143529:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768335.143535:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00100000:1.0:1775768335.143535:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076953600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768335.143542:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143548:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768335.143548:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768335.143554:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768335.143554:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143558:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768335.143563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768335.143567:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076954240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768335.143578:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768335.143581:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768335.143585:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768335.143587:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768335.143593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349640380 x1862023076953728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768335.143594:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143599:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143605:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143608:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076953728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768335.143611:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768335.143623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768335.143624:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076954368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768335.143631:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768335.143640:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768335.143646:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768335.143648:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768335.143649:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953856:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768335.143654:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4836 00000100:00080000:0.0:1775768335.143658:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768335.143661:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768335.143665:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768335.143676:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768335.143677:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076954496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768335.143687:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349642680 x1862023076953856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076953856/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768335.143717:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768335.143728:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349640e00 x1862023076953984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143740:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076953984/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143745:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143746:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143748:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076953984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768335.143751:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768335.143759:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143764:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349641500 x1862023076954112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076954112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143775:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143778:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143781:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768335.143785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768335.143801:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349642a00 x1862023076954240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143821:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076954240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143829:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143831:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143833:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768335.143837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768335.143849:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143856:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349640700 x1862023076954368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076954368/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143874:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143876:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768335.143884:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768335.143901:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768335.143910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768335/real 1775768335] req@ffffa04349643b80 x1862023076954496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768335.143917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076954496/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768335.143923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768335.143926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768335.143939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076954496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768335.335126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768335.335126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768336.167138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768336.167240:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768336.359153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768336.359154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768337.191143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768337.191193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768337.383134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768337.383134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768338.215149:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768338.215226:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768338.407132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768338.407132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768339.239161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768339.239186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768339.431129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768339.431129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768340.263107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768340.263109:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263116:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768340.263120:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263127:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263136:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263142:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:2.0:1775768340.263160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768340.263180:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076954624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768340.263203:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263206:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768340.263208:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263213:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263218:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263222:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076954624:192.168.202.158@tcp:250:kworker.0 00000008:00100000:0.0:1775768340.263231:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768340.263243:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076954752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768340.263250:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263252:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768340.263253:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263256:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263271:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263274:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768340.263277:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768340.263285:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076954880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768340.263292:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263294:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768340.263295:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263297:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263299:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263302:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263306:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768340.263308:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349643100 x1862023076954624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768340.263325:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076955008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768340.263326:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076954624/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768340.263345:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768340.263346:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263349:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263352:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076954624:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768340.263358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768340.263360:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263366:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263369:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263372:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768340.263374:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076954752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768340.263385:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076955136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768340.263392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768340.263392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768340.263396:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768340.263398:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263401:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263405:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768340.263405:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349643800 x1862023076954752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768340.263410:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263417:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768340.263422:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076954752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768340.263426:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076955264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768340.263432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076954752:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768340.263449:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263468:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768340.263469:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768340.263470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076954880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768340.263474:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263478:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263480:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263484:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768340.263491:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768340.263495:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076955392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768340.263501:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349641880 x1862023076954880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768340.263504:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263510:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768340.263513:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768340.263520:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263524:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263528:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263531:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768340.263531:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076954880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263538:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768340.263539:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076955520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768340.263545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263551:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076954880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768340.263552:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768340.263557:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768340.263559:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768340.263560:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955008:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768340.263563:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4841 00000100:00080000:0.0:1775768340.263566:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768340.263568:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768340.263572:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768340.263577:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768340.263580:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076955648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768340.263585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349641c00 x1862023076955008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263595:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076955008/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263599:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263601:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768340.263616:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768340.263625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263630:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349641180 x1862023076955136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263635:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076955136/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768340.263650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768340.263657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263662:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349640a80 x1862023076955264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076955264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263672:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263674:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263675:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768340.263678:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768340.263694:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263699:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349643100 x1862023076955392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263706:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076955392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263710:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263711:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263713:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768340.263716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768340.263725:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263732:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349643b80 x1862023076955520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076955520/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263751:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263754:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768340.263759:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768340.263777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768340.263786:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768340/real 1775768340] req@ffffa04349640700 x1862023076955648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768340.263795:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076955648/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768340.263800:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768340.263802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768340.263804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076955648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768340.455150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768340.455204:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768341.287155:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768341.287232:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768341.479143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768341.479143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768342.311139:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768342.311252:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768342.503134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768342.503134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768343.335166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768343.335256:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768343.527113:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768343.527160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768344.359195:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768344.359654:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768344.551132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768344.551636:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768345.383193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768345.383333:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768345.384142:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384149:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768345.384153:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384161:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384170:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384176:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768345.384208:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076955776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768345.384232:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384236:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768345.384237:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384241:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384244:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384248:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384257:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768345.384271:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076955904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768345.384282:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076955776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768345.384288:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384293:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768345.384295:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384299:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384302:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384306:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384312:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768345.384324:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076956032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768345.384333:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384336:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768345.384338:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384341:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384346:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384350:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384354:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768345.384362:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076956160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768345.384370:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384373:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768345.384376:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768345.384384:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768345.384388:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384394:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384398:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384420:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768345.384430:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076956288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768345.384441:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384445:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768345.384447:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384451:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384455:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384459:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384463:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768345.384475:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076956416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768345.384484:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768345.384488:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7fb80 x1862023076955776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768345.384489:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768345.384492:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384501:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384513:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384523:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384528:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768345.384540:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076956544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768345.384549:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076955776/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768345.384553:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384564:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768345.384568:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384582:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384587:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768345.384589:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768345.384594:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384601:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768345.384614:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768345.384615:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076956672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768345.384630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076955776:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768345.384635:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768345.384643:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768345.384646:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768345.384653:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4847 00000100:00080000:2.0:1775768345.384656:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768345.384660:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768345.384664:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768345.384676:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768345.384678:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076956800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768345.384716:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.384736:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7ce00 x1862023076955904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.384750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076955904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.384760:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.384764:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.384767:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076955904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768345.384775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956032:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768345.384797:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.384807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7c700 x1862023076956032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.384820:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076956032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.384828:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.384832:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.384835:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768345.384841:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768345.384862:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.384872:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7f800 x1862023076956160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.384883:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076956160/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.384891:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.384895:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.384898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768345.384904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768345.384936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.384946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7ea00 x1862023076956288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.384958:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076956288/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.384973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.384976:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.384980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076956288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768345.384986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956416:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768345.385006:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.385018:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7d880 x1862023076956416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.385067:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076956416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.385077:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.385080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.385084:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768345.385090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768345.385111:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.385122:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7ed80 x1862023076956544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.385133:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076956544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.385141:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.385144:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.385148:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768345.385155:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768345.385173:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.385183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7df80 x1862023076956672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.385194:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076956672/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.385201:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.385205:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.385208:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768345.385229:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768345.385247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768345.385257:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768345/real 1775768345] req@ffffa0434aa7e680 x1862023076956800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768345.385269:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076956800/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768345.385277:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768345.385280:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768345.385283:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076956800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768345.575165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768345.575165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768346.407194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768346.407222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768346.599127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768346.601563:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768347.431472:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768347.431659:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768347.623134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768347.623240:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768348.455118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768348.455176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768348.647164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768348.647295:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768349.479129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768349.479168:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768349.671139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768349.671139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768350.503177:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1775768350.503179:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768350.503187:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768350.503192:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.503202:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.503212:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.503218:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768350.503265:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076956928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.503291:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.503295:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768350.503298:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.503302:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.503306:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.503310:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.503319:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768350.503333:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076956928:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768350.503334:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076957056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768350.503342:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768350.503363:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.503368:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768350.503370:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.503374:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.503379:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.503383:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.503388:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768350.503411:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.503448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349642a00 x1862023076956928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.503465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076956928/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.503475:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.503479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.503483:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076956928:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768350.503504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768350.503525:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.503536:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349641500 x1862023076957056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.503548:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076957056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.503556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.503559:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.503562:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768350.505486:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076957184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505515:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505519:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768350.505521:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505526:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505531:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505535:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505542:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768350.505558:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076957312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505567:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505570:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768350.505571:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505574:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505578:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505581:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505585:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768350.505594:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076957440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505601:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505604:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768350.505605:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505608:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505611:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505615:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505619:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768350.505627:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076957568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505634:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505637:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768350.505638:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505642:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505645:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505647:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505651:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768350.505660:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076957696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505667:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505670:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768350.505672:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505676:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505679:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505682:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505686:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768350.505696:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076957824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768350.505704:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768350.505707:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768350.505709:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768350.505712:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4852 00000100:00080000:0.0:1775768350.505715:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768350.505718:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768350.505722:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768350.505731:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076957952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768350.505778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768350.505804:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349640e00 x1862023076957184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505829:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076957184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.505835:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.505837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.505839:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768350.505842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768350.505853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349642680 x1862023076957312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505865:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076957312/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.505869:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.505871:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.505873:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768350.505876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768350.505884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349640380 x1862023076957440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076957440/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.505902:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.505904:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.505906:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076957440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768350.505908:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768350.505917:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349642300 x1862023076957568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076957568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.505932:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.505933:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.505935:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768350.505937:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768350.505945:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349641500 x1862023076957696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505956:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076957696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.505960:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.505961:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.505963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768350.505966:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768350.505978:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.505985:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349642a00 x1862023076957824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.505995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076957824/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.506002:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.506004:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.506007:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768350.506013:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768350.506046:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768350.506055:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768350/real 1775768350] req@ffffa04349640700 x1862023076957952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768350.506062:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076957952/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768350.506067:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768350.506069:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768350.506070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076957952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768350.695127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768350.695503:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768351.527368:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768351.529554:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768351.719155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768351.719159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768352.551194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768352.551450:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768352.743141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768352.743433:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768353.575119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768353.575238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768353.767104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768353.767631:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768354.599290:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768354.599518:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768354.791138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768354.791138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768355.623163:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623171:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768355.623175:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623185:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623195:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623202:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768355.623206:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768355.623235:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775768355.623264:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076958080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768355.623291:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623294:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768355.623296:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623300:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623312:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623317:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768355.623338:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076958208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768355.623341:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076958080:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768355.623355:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768355.623359:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623363:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623366:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623369:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623374:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768355.623385:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076958336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768355.623394:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768355.623399:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623402:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623406:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623408:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768355.623413:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768355.623427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076958464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768355.623435:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623438:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768355.623440:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00100000:3.0:1775768355.623446:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350930700 x1862023076958080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768355.623448:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623455:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623463:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768355.623467:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076958080/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768355.623477:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076958592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768355.623480:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623487:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076958080:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768355.623497:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623500:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768355.623502:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623506:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623509:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768355.623524:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958208:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768355.623532:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768355.623547:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623554:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768355.623562:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350931880 x1862023076958208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768355.623568:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076958720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768355.623582:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768355.623585:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076958208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768355.623587:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768355.623590:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768355.623597:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623600:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768355.623608:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768355.623623:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:1.0:1775768355.623628:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768355.623628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768355.623634:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623640:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768355.623642:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350932a00 x1862023076958336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768355.623653:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076958848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768355.623658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076958336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768355.623670:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623678:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768355.623681:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768355.623683:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768355.623687:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00080000:3.0:1775768355.623689:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768355.623692:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768355.623693:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958336:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768355.623699:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768355.623701:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958464:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775768355.623706:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768355.623719:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076958976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768355.623730:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768355.623737:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768355.623741:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768355.623745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350932300 x1862023076958464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.623756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076958464/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623770:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958464:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768355.623771:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768355.623780:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4857 00000100:00100000:3.0:1775768355.623781:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958592:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768355.623784:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768355.623790:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768355.623795:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768355.623804:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768355.623810:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076959104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768355.623815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350930e00 x1862023076958592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.623831:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076958592/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623837:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623839:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623842:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076958592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768355.623857:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768355.623868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768355.623874:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350930000 x1862023076958720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.623880:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076958720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623885:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623886:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768355.623890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768355.623899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768355.623903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350930a80 x1862023076958848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.623909:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076958848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623913:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623914:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768355.623918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768355.623937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768355.623946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350931c00 x1862023076958976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.623956:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076958976/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.623963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.623966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.623969:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076958976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768355.623974:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076959104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768355.623989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768355.623998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768355/real 1775768355] req@ffffa04350932680 x1862023076959104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768355.624008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076959104/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768355.624015:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768355.624017:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768355.624020:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076959104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768355.815174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768355.815467:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768356.647154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768356.647164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768356.839143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768356.839767:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768357.671209:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768357.671228:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768357.863138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768357.863159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768358.695232:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768358.695235:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768358.887186:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768358.889445:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768359.720493:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768359.720747:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768359.911193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768359.911193:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768360.743162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768360.743235:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768360.743293:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743312:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768360.743318:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743330:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743343:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743350:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768360.743404:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076959232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768360.743438:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743441:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768360.743442:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743445:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743448:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743452:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743463:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768360.743482:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076959360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768360.743495:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076959232:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768360.743496:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743507:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768360.743509:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743513:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743517:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743521:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743526:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768360.743535:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076959488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768360.743542:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743545:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768360.743546:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743548:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743553:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743555:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743558:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768360.743565:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076959616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768360.743571:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743574:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768360.743578:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743584:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743588:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768360.743590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768360.743594:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743601:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768360.743616:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076959744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768360.743626:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743631:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768360.743633:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743643:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00100000:3.0:1775768360.743643:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349643b80 x1862023076959232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768360.743653:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743664:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743671:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768360.743674:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076959232/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.743686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.743693:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768360.743703:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076959872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768360.743704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076959232:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768360.743726:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743730:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768360.743731:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768360.743737:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743742:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743746:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743752:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768360.743753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768360.743769:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076960000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768360.743782:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768360.743786:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768360.743788:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768360.743794:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768360.743799:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00080000:0.0:1775768360.743805:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743810:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768360.743816:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768360.743817:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349643100 x1862023076959360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768360.743836:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076960128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768360.743841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076959360/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768360.743882:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768360.743884:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768360.743887:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768360.743890:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768360.743891:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768360.743903:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4862 00000100:00100000:3.0:1775768360.743905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959360:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768360.743914:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768360.743923:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768360.743923:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959488:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768360.743936:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768360.743953:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076960256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768360.743969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.743984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349640a80 x1862023076959488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.743997:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076959488/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744012:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744016:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768360.744023:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768360.744067:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744078:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349641180 x1862023076959616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076959616/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768360.744110:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959744:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768360.744127:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744137:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349641c00 x1862023076959744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744148:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076959744/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744156:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744159:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744162:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076959744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768360.744167:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076959872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768360.744187:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744197:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349641880 x1862023076959872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744209:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076959872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744221:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744224:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076959872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768360.744247:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768360.744285:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744296:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349643800 x1862023076960000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744307:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076960000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744314:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744317:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744321:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768360.744326:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768360.744343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744353:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349643b80 x1862023076960128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744364:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076960128/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744371:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744374:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744377:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768360.744382:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768360.744399:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768360.744408:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768360/real 1775768360] req@ffffa04349640700 x1862023076960256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768360.744418:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076960256/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768360.744425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768360.744428:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768360.744431:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076960256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768360.935135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768360.935174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768361.767140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768361.767179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768361.959145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768361.959145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768362.791199:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768362.791305:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768362.983206:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768362.983819:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768363.815250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768363.815361:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768364.007208:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768364.007208:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768364.839182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768364.839188:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768365.031328:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768365.031328:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768365.863141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768365.863179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768365.863242:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863246:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768365.863249:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863256:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863265:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863270:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768365.863297:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076960384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863314:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863317:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768365.863318:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863321:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863323:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863326:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863341:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768365.863366:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076960512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863373:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863376:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768365.863377:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863380:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863383:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863385:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863388:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768365.863397:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076960640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863403:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863406:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768365.863407:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863408:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863416:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863419:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863422:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768365.863429:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076960768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863438:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863441:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768365.863443:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768365.863445:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076960384:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768365.863448:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863455:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863465:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863471:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768365.863484:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076960896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863500:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863505:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768365.863506:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863516:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863521:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863532:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768365.863535:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768365.863563:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768365.863590:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768365.863610:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960896:192.168.202.158@tcp:8:kworker.0 00000008:00100000:3.0:1775768365.863645:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768365.863663:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076961024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863674:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863677:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768365.863679:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863683:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863702:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863708:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768365.863710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961024:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768365.863715:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768365.863730:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076961152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863742:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863745:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775768365.863748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961152:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768365.863896:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863900:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863904:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863907:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863912:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768365.863931:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076961280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768365.863967:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768365.863971:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768365.863973:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768365.863977:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4867 00000100:00080000:3.0:1775768365.863980:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768365.863984:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768365.863989:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768365.864001:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076961408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768365.864003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768365.864061:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768366.055176:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768366.055353:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768366.887135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768366.887175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768367.079136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768367.079136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768367.911116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768367.911149:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768368.103124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768368.103124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768368.935137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768368.935172:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775768368.939355:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775768368.939411:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768368.939428:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939468:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939477:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939483:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939490:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939495:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939505:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768368.939513:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775768368.939515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825d880 x1862023076961408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775768368.939520:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775768368.939544:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076961408/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939556:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939560:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939564:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768368.939571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825dc00 x1862023076961280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939579:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076961280/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939588:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768368.939593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825d500 x1862023076961152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939600:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076961152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939604:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939607:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939610:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768368.939623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825e300 x1862023076961024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076961024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939647:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939650:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076961024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768368.939656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825e680 x1862023076960768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076960768/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939681:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768368.939686:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825f100 x1862023076960896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939697:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076960896/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939708:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939710:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768368.939715:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825ea00 x1862023076960384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076960384/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939731:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076960384:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768368.939750:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825ed80 x1862023076960512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939761:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076960512/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939769:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939772:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768368.939781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768365/real 1775768368] req@ffffa0435825c000 x1862023076960640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768368.939791:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076960640/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768368.939797:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768368.939800:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768368.939802:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076960640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768369.127125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768369.127125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768369.959181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768369.959192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768370.151132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768370.151134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768370.983290:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768370.983290:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983303:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768370.983308:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983319:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983333:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983342:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768370.983383:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076961536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983413:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983417:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768370.983419:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983423:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983428:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983432:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983444:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768370.983448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076961536:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768370.983462:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076961664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983473:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983476:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768370.983478:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983482:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983486:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983489:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983494:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768370.983505:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076961792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983514:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983517:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768370.983519:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983524:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4872 00000800:00000100:1.0:1775768370.983528:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768370.983531:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983536:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983543:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768370.983555:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076961920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983564:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983567:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768370.983569:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983574:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00100000:1.0:1775768370.983578:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825ce00 x1862023076961536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768370.983579:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983586:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983595:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768370.983607:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076961536/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768370.983616:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076962048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768370.983621:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.983628:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.983632:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076961536:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768370.983640:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983644:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768370.983646:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983651:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983655:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983660:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768370.983664:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961664:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768370.983666:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768370.983683:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076962176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983693:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983696:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768370.983697:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983701:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983704:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983707:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983712:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768370.983723:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076962304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983731:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983734:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768370.983735:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983739:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983742:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983746:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983750:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768370.983759:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076962432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768370.983766:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768370.983770:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768370.983771:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768370.983775:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4872 00000100:00080000:3.0:1775768370.983778:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768370.983781:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768370.983786:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768370.983794:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076962560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1775768370.983860:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1775768370.983962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.983976:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825c000 x1862023076961664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.983996:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076961664/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768370.984019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768370.984058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825ed80 x1862023076961792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984081:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076961792/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984090:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984094:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984097:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768370.984103:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768370.984121:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984130:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825ea00 x1862023076961920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984141:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076961920/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984175:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076961920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768370.984180:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768370.984199:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984208:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825f100 x1862023076962048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984218:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076962048/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984230:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984232:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984235:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768370.984253:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768370.984271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825e680 x1862023076962176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984290:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076962176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984299:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984301:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984305:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768370.984310:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768370.984328:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984337:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825e300 x1862023076962304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984345:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076962304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984352:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984357:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768370.984362:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768370.984378:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984387:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825d500 x1862023076962432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076962432/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984404:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984407:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984409:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768370.984414:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768370.984431:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768370.984440:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768370/real 1775768370] req@ffffa0435825dc00 x1862023076962560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768370.984451:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076962560/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768370.984460:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768370.984462:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768370.984465:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076962560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768371.175180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768371.175243:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768372.007265:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768372.007721:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768372.199191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768372.199192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768373.031204:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768373.031252:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768373.223155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768373.223155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768374.055160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768374.055259:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768374.247134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768374.247147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768375.079198:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768375.079227:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768375.271142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768375.271142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768376.103174:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768376.103194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768376.103202:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103208:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768376.103211:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103221:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103231:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768376.103237:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768376.103280:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076962688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768376.103307:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103311:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768376.103313:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103317:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103320:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768376.103325:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768376.103334:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768376.103346:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076962688:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768376.103353:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076962816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768376.103365:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103368:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768376.103370:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103374:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103378:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768376.103382:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768376.103387:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768376.103400:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076962944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768376.103405:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768376.103415:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103419:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768376.103420:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103425:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103440:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768376.103442:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350932680 x1862023076962688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103447:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768376.103457:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768376.103465:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076962688/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768376.103471:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076963072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768376.103481:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.103486:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.103493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076962688:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768376.103499:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103503:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768376.103505:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103510:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103514:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768376.103516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962816:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768376.103521:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768376.103534:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775768376.103539:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768376.103551:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076963200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768376.103553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350931c00 x1862023076962816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103580:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768376.103584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076962816/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103585:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768376.103589:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103597:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:3.0:1775768376.103602:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768376.103603:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768376.103610:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768376.103612:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768376.103617:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962816:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775768376.103619:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768376.103624:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768376.103633:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076963328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768376.103643:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768376.103644:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103648:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768376.103650:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768376.103654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350930a80 x1862023076962944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103656:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103663:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768376.103672:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076962944/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768376.103678:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768376.103684:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768376.103684:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.103691:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076962944:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775768376.103694:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768376.103699:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768376.103709:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076963456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768376.103719:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768376.103720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768376.103733:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350930000 x1862023076963072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103741:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768376.103743:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076963072/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768376.103744:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103753:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103760:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768376.103767:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768376.103768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768376.103775:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768376.103775:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.103779:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768376.103791:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076963584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768376.103798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963200:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768376.103806:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768376.103810:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768376.103812:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768376.103817:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4877 00000100:00080000:1.0:1775768376.103822:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768376.103825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768376.103838:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768376.103840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350930e00 x1862023076963200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768376.103847:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768376.103854:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076963200/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768376.103863:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076963712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768376.103864:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.103872:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.103879:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768376.103887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768376.103910:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768376.103924:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350932300 x1862023076963328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768376.103934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076963328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768376.103941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.103944:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.103947:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768376.103952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768376.103969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768376.103979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350932a00 x1862023076963456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768376.103988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076963456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768376.103996:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.103999:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.104001:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768376.104016:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768376.104053:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768376.104064:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350931880 x1862023076963584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768376.104074:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076963584/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768376.104081:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.104085:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.104087:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768376.104093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768376.104127:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768376.104136:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768376/real 1775768376] req@ffffa04350930700 x1862023076963712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768376.104143:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076963712/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768376.104148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768376.104150:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768376.104152:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076963712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768376.295120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768376.295120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768377.127317:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768377.127370:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768377.319336:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768377.319489:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768378.151225:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768378.152619:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768378.343212:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768378.343213:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768379.175153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768379.175215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768379.367203:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768379.367240:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768380.199147:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768380.199186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768380.391179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768380.391880:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768381.223248:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768381.223542:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.223549:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768381.223553:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.223563:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.223571:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.223577:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768381.223611:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076963840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768381.223633:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.223637:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768381.223638:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.223643:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.223647:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.223651:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768381.223660:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768381.223662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076963840:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768381.223672:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076963968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768381.223683:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.223686:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768381.223688:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.223692:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4882 00000800:00000100:3.0:1775768381.223714:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768381.223740:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768381.223867:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7c000 x1862023076963840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.223884:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076963840/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.223894:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.223898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.223902:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076963840:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768381.223922:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963968:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768381.223977:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.223983:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768381.223989:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768381.224012:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076964096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768381.224022:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224042:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768381.224045:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224049:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.224053:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.224065:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768381.224070:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768381.224081:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076964224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768381.224091:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224094:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768381.224096:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224100:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.224104:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.224108:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768381.224112:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768381.224115:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768381.224128:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076964352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768381.224132:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7e300 x1862023076963968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768381.224146:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224152:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768381.224153:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224159:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00100000:3.0:1775768381.224159:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076963968/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768381.224165:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.224172:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768381.224175:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768381.224178:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768381.224182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076963968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768381.224203:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076964480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768381.224203:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964096:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768381.224219:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224222:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768381.224224:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224228:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.224233:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768381.224233:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768381.224240:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768381.224246:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768381.224251:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7d500 x1862023076964096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768381.224269:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076964608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768381.224269:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076964096/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768381.224281:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224284:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768381.224286:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224290:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.224293:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768381.224297:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768381.224302:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768381.224311:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076964736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768381.224320:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768381.224324:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768381.224326:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768381.224334:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4882 00000100:00080000:2.0:1775768381.224340:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768381.224341:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768381.224345:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768381.224348:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768381.224353:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768381.224353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768381.224364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768381.224369:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076964864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768381.224400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224412:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7dc00 x1862023076964224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224425:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076964224/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768381.224444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768381.224461:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224470:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7f100 x1862023076964352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224479:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076964352/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224485:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224488:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076964352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768381.224503:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768381.224520:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224529:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7f480 x1862023076964480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076964480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224544:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224547:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768381.224554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768381.224568:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224576:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7c380 x1862023076964608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076964608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224594:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768381.224599:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768381.224612:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224621:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7e300 x1862023076964736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224631:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076964736/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224637:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224640:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768381.224646:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768381.224661:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768381.224670:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768381/real 1775768381] req@ffffa0434aa7c000 x1862023076964864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768381.224680:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076964864/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768381.224688:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768381.224690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768381.224705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076964864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768381.415136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768381.415239:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768382.247194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768382.247307:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768382.439167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768382.443218:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768383.271248:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768383.272087:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768383.463249:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768383.463463:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768384.295169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768384.295325:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768384.487142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768384.487142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768385.319222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768385.319551:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768385.511154:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768385.511154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768386.343127:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343137:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775768386.343138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768386.343141:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343152:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4888 00010000:00010000:3.0:1775768386.343161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768386.343162:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343170:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768386.343206:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076964992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343246:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343250:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768386.343251:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343255:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343259:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343263:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343272:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768386.343285:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076965120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343295:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343298:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768386.343300:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343303:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343307:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343311:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343316:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768386.343326:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076965248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343335:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343339:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768386.343340:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343344:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343351:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768386.343367:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076965376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343377:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343381:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768386.343381:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076964992:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768386.343384:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343392:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343395:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343399:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343403:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768386.343416:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076965504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343425:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343428:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768386.343430:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343433:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343437:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343441:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343446:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768386.343452:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768386.343458:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076965632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343467:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343471:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768386.343473:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343477:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343480:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343484:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343489:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768386.343498:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076965760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343506:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343509:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768386.343510:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343514:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343516:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343519:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768386.343532:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076965888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768386.343540:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768386.343543:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768386.343545:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768386.343548:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4888 00000100:00080000:1.0:1775768386.343551:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768386.343554:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768386.343558:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768386.343567:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076966016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768386.344271:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350930700 x1862023076964992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.344291:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076964992/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.344981:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.344987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.344991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076964992:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768386.345208:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768386.345240:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345265:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350931880 x1862023076965120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345278:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076965120/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345286:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345292:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768386.345298:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965248:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768386.345316:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345326:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350932a00 x1862023076965248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345337:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076965248/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345345:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345348:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345350:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768386.345374:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768386.345393:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345402:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350932300 x1862023076965376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076965376/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345419:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345422:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768386.345430:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768386.345446:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345456:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350930e00 x1862023076965504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076965504/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345473:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345476:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345491:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076965504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768386.345498:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768386.345516:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345526:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350930000 x1862023076965632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345765:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076965632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345773:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345775:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768386.345779:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768386.345794:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350930a80 x1862023076965760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345808:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076965760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345812:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345814:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345816:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768386.345821:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768386.345838:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345848:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350931c00 x1862023076965888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076965888/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.345926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.345929:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076965888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768386.345935:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768386.345968:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768386.345979:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768386/real 1775768386] req@ffffa04350932680 x1862023076966016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768386.345990:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076966016/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768386.345998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768386.346001:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768386.346004:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768386.535146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768386.536045:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768387.367126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768387.367147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768387.559136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768387.559389:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768388.391121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768388.391146:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768388.583109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768388.583112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768389.415220:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768389.415531:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768389.607159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768389.608092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768390.439511:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768390.440447:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768390.631123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768390.632053:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768391.463137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768391.463161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768391.463379:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.463530:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768391.463535:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.463569:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.463578:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.463594:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768391.463665:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076966144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.463722:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.463807:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768391.463809:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.463814:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.463819:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768391.463827:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076966144:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768391.463884:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768391.463919:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768391.463923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349642a00 x1862023076966144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768391.463941:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768391.463943:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076966144/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.463955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.463970:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.463974:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076966144:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768391.463994:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076966272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768391.464057:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768391.464083:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.464097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349641500 x1862023076966272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.464110:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076966272/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.464119:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.464122:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.464126:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966272:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768391.464257:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.464261:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768391.464264:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.464268:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.464273:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.464336:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.464342:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768391.464356:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076966400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.464379:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.464383:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768391.464385:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.464388:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.464392:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.464396:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.464401:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768391.464411:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076966528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768391.464668:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768391.464729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768391.464828:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768391.464849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349642300 x1862023076966400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.464863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076966400/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.464871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.464875:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.464878:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966400:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768391.464892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768391.464915:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.464927:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349640380 x1862023076966528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.464938:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076966528/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.464946:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.464949:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.464952:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966528:192.168.202.158@tcp:8:kworker.0 00000100:00080000:0.0:1775768391.465097:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768391.465099:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.465104:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.465107:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.465110:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.465123:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768391.465133:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076966656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.465152:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.465154:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768391.465155:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.465158:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.465160:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.465162:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.465165:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768391.465181:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076966784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.465186:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.465188:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768391.465190:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.465192:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.465194:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.465195:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.465209:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768391.465218:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076966912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.465238:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.465241:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768391.465243:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.465246:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.465250:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.465267:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.465271:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768391.465279:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076967040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768391.465294:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768391.465296:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768391.465297:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768391.465300:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4893 00000100:00080000:0.0:1775768391.465302:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768391.465304:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768391.465307:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768391.465322:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076967168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768391.465412:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768391.465435:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.465463:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349642680 x1862023076966656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.465473:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076966656/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.465479:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.465481:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.465483:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076966656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768391.465494:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768391.465518:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.465535:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349640e00 x1862023076966784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.465561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076966784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.465570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.465583:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.465587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768391.465591:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768391.465623:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.465650:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349642300 x1862023076966912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.465663:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076966912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.465669:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.465671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.465673:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076966912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768391.465679:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768391.465698:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.465725:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349641500 x1862023076967040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.465756:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076967040/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.465765:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.465768:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.465784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768391.465790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768391.465825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768391.465980:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768391/real 1775768391] req@ffffa04349642a00 x1862023076967168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768391.466009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076967168/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768391.466060:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768391.466064:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768391.466067:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768391.655101:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768391.656016:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768392.487210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768392.489803:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768392.679163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768392.680008:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768393.511126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768393.511175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768393.703213:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768393.703831:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768394.535126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768394.535486:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768394.727182:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768394.727184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768395.559203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768395.559262:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768395.751093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768395.751226:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768396.583136:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768396.583142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768396.583146:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768396.583151:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583158:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583169:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583174:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768396.583204:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076967296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583224:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583226:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768396.583227:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583230:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583231:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583234:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583240:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768396.583247:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076967424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583252:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583254:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768396.583255:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583257:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583259:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583260:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583262:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768396.583267:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076967552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583271:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583274:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768396.583275:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583277:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583278:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583280:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583283:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768396.583287:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076967680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583291:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583292:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768396.583293:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583295:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583296:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583298:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583300:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768396.583304:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076967808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583309:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583310:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768396.583311:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583313:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583314:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583316:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583318:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768396.583323:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076967936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583327:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583329:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768396.583330:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583331:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583333:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583334:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583336:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768396.583340:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076968064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583344:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583346:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768396.583346:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583348:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583349:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583351:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583353:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768396.583357:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076968192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768396.583361:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768396.583363:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768396.583364:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768396.583366:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4898 00000100:00080000:0.0:1775768396.583367:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768396.583369:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768396.583370:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768396.583376:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076968320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768396.583462:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076967296:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768396.583515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.583553:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349640700 x1862023076967296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.583568:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076967296/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.583577:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.583581:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.583584:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076967296:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768396.583590:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768396.585814:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.585836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349643b80 x1862023076967424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.585852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076967424/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.585862:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.585865:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.585868:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768396.585876:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967552:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768396.585899:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.585921:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349643800 x1862023076967552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.585934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076967552/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.585941:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.585945:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.585948:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768396.585953:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768396.585974:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.585984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349641880 x1862023076967680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.585995:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076967680/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586007:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586010:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768396.586018:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768396.586053:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.586064:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349641c00 x1862023076967808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.586072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076967808/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586077:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586080:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586082:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076967808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768396.586086:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768396.586101:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.586110:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349641180 x1862023076967936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.586121:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076967936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586129:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586132:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076967936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768396.586142:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768396.586158:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.586167:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349640a80 x1862023076968064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.586176:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076968064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586184:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586186:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586188:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768396.586192:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768396.586203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.586209:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349643100 x1862023076968192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.586215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076968192/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586220:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586223:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768396.586226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768396.586234:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768396.586240:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768396/real 1775768396] req@ffffa04349642a00 x1862023076968320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768396.586249:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076968320/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768396.586255:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768396.586258:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768396.586260:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076968320:192.168.202.158@tcp:8:kworker.0 00010000:00010000:0.0:1775768396.586378:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768396.775133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768396.775133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768397.607133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768397.607376:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768397.799128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768397.799133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768398.631130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768398.631188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768398.823125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768398.823127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768399.656368:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768399.660052:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768399.847157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768399.847639:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768400.679143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768400.679269:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768400.871142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768400.871142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768401.703163:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703173:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768401.703177:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768401.703177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768401.703177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768401.703188:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703197:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703204:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768401.703236:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076968448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768401.703259:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703263:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768401.703265:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768401.703269:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703273:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703277:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703286:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768401.703294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076968448:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768401.703299:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076968576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768401.703310:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703313:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768401.703315:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768401.703319:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703322:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703326:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703330:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768401.703340:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076968704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768401.703348:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703351:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768401.703352:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768401.703358:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768401.703361:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703365:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703368:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703373:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768401.703385:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076968832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768401.703394:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703398:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768401.703399:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768401.703404:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349641500 x1862023076968448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768401.703405:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703410:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768401.703423:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076968448/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768401.703426:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703434:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775768401.703437:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.703443:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768401.703447:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076968960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768401.703448:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076968448:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768401.703462:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703465:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768401.703467:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768401.703471:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703474:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768401.703478:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968576:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768401.703480:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703488:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768401.703499:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076969088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768401.703507:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768401.703512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768401.703513:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768401.703516:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768401.703522:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703526:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703530:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768401.703531:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349642300 x1862023076968576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768401.703537:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768401.703553:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076969216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768401.703554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076968576/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.703581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768401.703583:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768401.703584:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.703589:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968576:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768401.703590:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768401.703595:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768401.703598:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768401.703603:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:0.0:1775768401.703607:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768401.703611:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703617:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768401.703618:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.703628:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349640e00 x1862023076968704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768401.703632:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076969344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768401.703642:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076968704/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768401.703647:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768401.703655:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768401.703659:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768401.703660:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768401.703664:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4903 00000100:00080000:3.0:1775768401.703666:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768401.703669:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768401.703672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968704:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768401.703676:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768401.703683:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768401.703686:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768401.703693:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076969472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768401.703712:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.703724:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349642680 x1862023076968832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.703742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076968832/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.703749:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.703752:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.703754:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768401.703761:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768401.703793:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.703804:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349640380 x1862023076968960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.703815:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076968960/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.703822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.703824:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.703827:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076968960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768401.703848:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768401.703888:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.703901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349641500 x1862023076969088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.703912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076969088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.703924:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.703927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.703930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768401.703936:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768401.703955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.704069:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349642a00 x1862023076969216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.704088:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076969216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.704194:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.704197:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.704201:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768401.704207:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768401.704267:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.704280:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349643100 x1862023076969344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.704288:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076969344/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.704293:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.704295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.704297:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768401.704301:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768401.704314:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768401.704319:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768401/real 1775768401] req@ffffa04349640a80 x1862023076969472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768401.704325:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076969472/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768401.704330:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768401.704331:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768401.704333:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076969472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768401.895128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768401.895128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768402.727152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768402.728167:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768402.919133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768402.919133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768403.751145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768403.752147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768403.943161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768403.943161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768404.775126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768404.775290:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768404.967125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768404.967131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768405.799127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768405.799153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768405.991133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768405.991133:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768406.823129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768406.823134:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823139:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768406.823143:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823153:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823162:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823169:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768406.823175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768406.823205:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076969600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768406.823228:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823231:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768406.823233:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823236:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823240:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823244:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823253:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768406.823265:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076969600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768406.823272:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076969728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768406.823283:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823286:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768406.823288:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823292:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823296:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823309:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823318:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768406.823331:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076969856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768406.823338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768406.823347:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823351:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768406.823353:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823358:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823363:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823367:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823372:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768406.823381:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349641180 x1862023076969600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768406.823385:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076969984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768406.823407:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823410:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768406.823413:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768406.823413:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076969600/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768406.823422:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823429:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768406.823432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768406.823434:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823442:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1775768406.823442:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823449:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076969600:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768406.823459:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076970112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768406.823469:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823471:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768406.823474:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823477:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823481:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768406.823485:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768406.823486:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823495:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768406.823509:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076970240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768406.823519:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768406.823520:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823524:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768406.823525:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823528:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823532:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823535:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823539:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768406.823546:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349641c00 x1862023076969728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768406.823550:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076970368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768406.823573:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823575:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768406.823577:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768406.823580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076969728/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768406.823582:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823586:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823590:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823597:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768406.823600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823606:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768406.823609:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076970496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768406.823613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969728:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768406.823626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768406.823629:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768406.823634:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768406.823635:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768406.823640:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4908 00000100:00080000:0.0:1775768406.823645:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768406.823649:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768406.823653:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768406.823655:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768406.823666:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076970624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768406.823675:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349641880 x1862023076969856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.823689:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076969856/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.823698:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823705:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969856:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768406.823711:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768406.823727:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.823737:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349643800 x1862023076969984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.823748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076969984/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.823755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076969984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768406.823777:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076970112:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768406.823792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.823801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349643b80 x1862023076970112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.823831:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076970112/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.823838:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823841:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823844:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076970112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768406.823861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768406.823878:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.823889:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349640700 x1862023076970240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.823905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076970240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.823913:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768406.823924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768406.823948:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.823958:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349641180 x1862023076970368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.823968:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076970368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.823975:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.823978:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.823981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768406.823986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768406.824001:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.824010:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349640a80 x1862023076970496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.824021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076970496/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.824051:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.824054:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.824056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768406.824061:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768406.824077:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768406.824085:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768406/real 1775768406] req@ffffa04349643100 x1862023076970624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768406.824094:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076970624/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768406.824100:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768406.824102:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768406.824104:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076970624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768407.015144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768407.015230:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768407.847117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768407.847145:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768408.039138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768408.039138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768408.871142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768408.871174:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768409.063127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768409.063606:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768409.895132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768409.895166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768410.087137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768410.087137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768410.919156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768410.919271:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768411.111129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768411.111129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:0.0:1775768411.943131:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768411.943132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768411.943141:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768411.943145:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943155:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943167:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775768411.943167:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768411.943175:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768411.943219:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076970752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943243:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943247:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768411.943249:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943252:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943255:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768411.943260:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943268:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768411.943277:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076970880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943285:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943288:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768411.943289:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943292:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943295:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768411.943299:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943302:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768411.943309:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076971008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943318:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943321:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768411.943322:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943326:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943329:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768411.943333:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076970752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768411.943335:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943343:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768411.943353:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076971136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943362:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943373:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768411.943375:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943379:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943383:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768411.943387:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943391:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768411.943403:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076971264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768411.943404:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768411.943417:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943421:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768411.943422:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943426:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943436:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768411.943440:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349642a00 x1862023076970752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768411.943441:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943448:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768411.943458:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076971392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943485:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943488:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768411.943491:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943497:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943501:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768411.943504:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943509:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768411.943514:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076970752/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768411.943521:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076971520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768411.943530:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768411.943535:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768411.943538:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768411.943540:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768411.943544:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768411.943547:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076970752:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768411.943551:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943556:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768411.943559:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076970880:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768411.943562:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943569:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768411.943580:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076971648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768411.943588:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768411.943592:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768411.943594:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768411.943597:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4913 00000100:00080000:0.0:1775768411.943601:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768411.943605:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768411.943605:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768411.943612:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768411.943622:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076971776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768411.943624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349641500 x1862023076970880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.943652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076970880/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.943660:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.943663:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.943667:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076970880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768411.943673:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768411.943691:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.943701:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349640380 x1862023076971008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.943713:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076971008/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.943735:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.943738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.943742:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768411.943964:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768411.943984:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.943994:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349642680 x1862023076971136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944005:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076971136/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944011:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944014:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944017:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768411.944022:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768411.944057:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.944066:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349640e00 x1862023076971264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944076:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076971264/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944083:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944085:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076971264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768411.944093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768411.944107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.944124:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349642300 x1862023076971392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944134:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076971392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944141:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944144:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944146:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768411.944151:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768411.944165:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.944173:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349643100 x1862023076971520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076971520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944189:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944191:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768411.944198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768411.944222:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.944235:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349640a80 x1862023076971648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944246:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076971648/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944253:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944256:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944258:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768411.944263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768411.944279:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768411.944289:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768411/real 1775768411] req@ffffa04349641180 x1862023076971776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768411.944300:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076971776/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768411.944305:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768411.944307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768411.944309:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076971776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768412.135124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768412.135124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768412.967128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768412.967180:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768413.159130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768413.159131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768413.991161:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768413.992085:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768414.183152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768414.183154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768415.015124:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768415.015224:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768415.207132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768415.207133:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768416.039133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768416.039181:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768416.231175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768416.233237:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768417.063082:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768417.063130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768417.063134:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768417.063145:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063153:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063162:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063169:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768417.063205:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076971904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768417.063232:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063236:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768417.063237:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063241:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063245:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063250:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063259:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768417.063260:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076971904:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768417.063274:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076972032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768417.063285:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063288:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768417.063290:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063297:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063300:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063304:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768417.063314:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076972160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768417.063325:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768417.063331:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063335:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768417.063337:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063342:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063347:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063351:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063357:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768417.063368:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349640700 x1862023076971904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768417.063370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076972288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768417.063396:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063399:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768417.063400:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063404:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063408:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063411:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063416:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768417.063417:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076971904/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768417.063430:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076972416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768417.063430:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063439:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063445:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076971904:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768417.063446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063452:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768417.063454:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063458:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063462:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063465:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063469:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972032:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768417.063471:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768417.063485:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076972544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768417.063492:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768417.063497:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768417.063500:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768417.063502:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768417.063506:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349643b80 x1862023076972032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768417.063508:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063513:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063518:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063525:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768417.063530:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076972032/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768417.063537:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076972672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768417.063543:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768417.063549:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768417.063550:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768417.063554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768417.063557:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768417.063558:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972032:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768417.063563:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063568:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768417.063570:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768417.063574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063579:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768417.063589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076972800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768417.063596:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768417.063606:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768417.063610:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349643800 x1862023076972160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768417.063612:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768417.063614:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768417.063621:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4918 00000100:00080000:0.0:1775768417.063626:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768417.063633:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768417.063641:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768417.063645:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076972160/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063652:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768417.063655:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076972928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768417.063657:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972160:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768417.063668:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972288:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768417.063685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063693:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349641880 x1862023076972288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.063701:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076972288/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063706:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063723:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768417.063733:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768417.063758:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349641c00 x1862023076972416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.063784:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076972416/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063799:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063806:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076972416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768417.063824:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768417.063855:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063865:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349640700 x1862023076972544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.063872:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076972544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063878:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063881:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063884:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768417.063889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768417.063907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349641180 x1862023076972672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.063927:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076972672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063934:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063937:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768417.063950:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768417.063963:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.063973:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349640a80 x1862023076972800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.063980:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076972800/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.063984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.063986:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.063987:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768417.063990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768417.063999:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768417.064004:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768417/real 1775768417] req@ffffa04349643100 x1862023076972928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768417.064010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076972928/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768417.064014:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768417.064015:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768417.064018:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076972928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768417.255098:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768417.255123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768418.087096:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768418.087114:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768418.279160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768418.279263:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768419.111265:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768419.111280:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768419.303104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768419.303104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768420.135126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768420.135175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768420.327139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768420.327139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768421.159152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768421.159216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768421.351125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768421.351125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:1.0:1775768422.183161:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768422.183163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768422.183177:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768422.183183:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183197:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00080000:1.0:1775768422.183210:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1775768422.183213:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768422.183220:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768422.183268:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076973056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768422.183302:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183306:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768422.183308:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183312:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00080000:1.0:1775768422.183316:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183320:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183332:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768422.183347:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076973056:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768422.183351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076973184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768422.183364:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183373:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768422.183375:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183379:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00080000:1.0:1775768422.183383:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183386:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183391:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768422.183403:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076973312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768422.183419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768422.183424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768422.183433:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768422.183441:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00080000:1.0:1775768422.183446:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183450:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183456:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768422.183468:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076973440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768422.183478:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183481:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768422.183483:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183487:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00100000:3.0:1775768422.183491:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350932680 x1862023076973056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768422.183492:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183500:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183509:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768422.183522:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076973056/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768422.183523:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076973568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768422.183545:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768422.183547:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183552:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768422.183555:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768422.183556:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768422.183562:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076973056:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768422.183563:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00080000:1.0:1775768422.183571:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183576:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183581:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768422.183592:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076973696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768422.183595:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768422.183610:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183614:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768422.183616:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183620:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4923 00000800:00000100:3.0:1775768422.183625:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768422.183626:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183631:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768422.183637:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768422.183641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350931c00 x1862023076973184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768422.183651:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076973824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768422.183658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076973184/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768422.183681:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768422.183684:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768422.183686:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768422.183690:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768422.183691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768422.183697:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00100000:3.0:1775768422.183698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973184:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768422.183702:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183710:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973312:192.168.202.158@tcp:38:kworker.0 00000008:00100000:1.0:1775768422.183718:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768422.183733:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076973952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768422.183741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183753:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350930a80 x1862023076973312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768422.183754:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768422.183761:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768422.183765:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768422.183773:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4923 00000100:00100000:3.0:1775768422.183777:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076973312/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768422.183778:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768422.183785:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768422.183789:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768422.183792:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768422.183795:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.183800:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768422.183808:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768422.183809:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076974080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768422.183831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183843:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350930000 x1862023076973440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.183856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076973440/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.183863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.183867:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.183870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768422.183875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768422.183893:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183901:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350930e00 x1862023076973568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.183908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076973568/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.183915:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.183917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.183919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076973568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768422.183929:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768422.183939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183945:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350932300 x1862023076973696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.183951:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076973696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.183956:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.183958:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.183960:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768422.183963:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768422.183977:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.183985:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350932a00 x1862023076973824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.184008:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076973824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.184047:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.184049:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.184052:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768422.184055:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768422.184070:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.184077:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350931880 x1862023076973952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.184083:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076973952/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.184088:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.184090:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.184092:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076973952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768422.184095:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768422.184103:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768422.184107:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768422/real 1775768422] req@ffffa04350930700 x1862023076974080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768422.184113:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076974080/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768422.184117:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768422.184119:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768422.184121:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768422.375178:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768422.375178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768423.207192:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768423.207216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768423.399166:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768423.399166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768424.231176:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768424.231255:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768424.423167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768424.423167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768425.255119:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768425.255171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768425.447165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768425.447170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768426.279182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768426.279257:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768426.471158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768426.471158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768427.303153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768427.303157:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303170:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768427.303177:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1775768427.303177:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768427.303194:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303207:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303216:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768427.303261:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076974208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303291:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303295:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768427.303297:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303300:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303304:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303307:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303317:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768427.303331:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076974336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768427.303333:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076974208:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768427.303341:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303352:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768427.303356:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303360:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303362:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303364:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303367:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768427.303374:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076974464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303379:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303382:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768427.303384:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303386:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303389:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303391:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303394:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768427.303400:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076974592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303406:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303408:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768427.303409:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303414:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303417:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303419:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303422:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768427.303429:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076974720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303433:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303436:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768427.303438:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768427.303447:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768427.303450:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303454:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303457:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303460:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768427.303467:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076974848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303472:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768427.303475:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768427.303477:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303480:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303485:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303486:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303489:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768427.303501:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076974976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768427.303509:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768427.303511:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349642300 x1862023076974208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768427.303517:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768427.303521:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303536:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303543:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303547:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303552:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768427.303554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076974208/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768427.303570:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076975104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768427.303571:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303586:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768427.303596:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768427.303600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076974208:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768427.303601:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768427.303607:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768427.303625:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4928 00000100:00080000:0.0:1775768427.303629:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768427.303632:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768427.303638:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768427.303649:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076975232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768427.303657:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768427.303685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349640e00 x1862023076974336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303712:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076974336/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303725:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303727:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768427.303731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974464:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768427.303748:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303756:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349642680 x1862023076974464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076974464/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303768:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303770:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303771:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768427.303775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768427.303786:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349640380 x1862023076974592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076974592/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768427.303807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974720:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768427.303816:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303820:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349641500 x1862023076974720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303827:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076974720/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303833:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303834:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076974720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768427.303837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768427.303849:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349642a00 x1862023076974848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303867:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076974848/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303875:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768427.303887:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974976:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768427.303904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349643100 x1862023076974976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076974976/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303939:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303942:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076974976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768427.303951:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076975104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768427.303969:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.303978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349640a80 x1862023076975104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.303985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076975104/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.303993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.303996:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.303998:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076975104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768427.304018:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076975232:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768427.304055:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768427.304065:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768427/real 1775768427] req@ffffa04349641180 x1862023076975232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768427.304076:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076975232/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768427.304089:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768427.304092:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768427.304095:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076975232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768427.495152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768427.495435:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768428.327181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768428.327260:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768428.519109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768428.519109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768429.351152:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768429.351205:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768429.543137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768429.543137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768430.375241:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768430.375273:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768430.567134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768430.567134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768431.399193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768431.399238:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768431.591131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768431.591137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768432.423184:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768432.423197:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423205:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768432.423209:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423219:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423230:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423236:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768432.423237:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768432.423270:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076975360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423297:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423301:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768432.423303:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423307:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423311:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423315:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423325:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768432.423340:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076975360:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768432.423341:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076975488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423368:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423372:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768432.423374:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423378:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423382:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423385:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423390:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768432.423400:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076975616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423405:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423408:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768432.423409:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423411:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423414:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423418:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423424:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768432.423434:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768432.423437:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076975744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423450:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423454:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768432.423457:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768432.423457:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975616:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768432.423465:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423471:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423477:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768432.423477:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975744:192.168.202.158@tcp:8:kworker.0 00000008:00100000:1.0:1775768432.423483:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768432.423497:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076975872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423510:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423514:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768432.423516:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423521:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00100000:3.0:1775768432.423523:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975872:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768432.423527:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423543:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423550:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768432.423587:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076976000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423604:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423606:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768432.423608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423616:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423620:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423622:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423626:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768432.423636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076976128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423641:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423643:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768432.423644:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423646:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423647:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423649:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423652:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768432.423657:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076976256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768432.423662:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768432.423664:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768432.423665:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768432.423667:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4934 00000100:00080000:1.0:1775768432.423669:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768432.423671:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768432.423674:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768432.423681:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076976384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768432.423775:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768432.423802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768432.423819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768432.423836:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768432.615164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768432.615288:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768433.447400:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768433.447437:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768433.639131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768433.639131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768434.471125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768434.471144:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768434.663168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768434.663169:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768435.495187:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775768435.495364:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775768435.495412:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768435.495431:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495483:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495498:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495506:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495514:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495523:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495530:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495563:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768435.495572:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:3.0:1775768435.495691:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350930000 x1862023076976000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1775768435.495692:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768435.495720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076976000/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495732:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768435.495751:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350930a80 x1862023076976128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076976128/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495770:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495777:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768435.495784:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350931c00 x1862023076976256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495796:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076976256/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495804:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495808:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495811:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768435.495818:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350932680 x1862023076976384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495830:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076976384/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495839:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495842:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495845:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076976384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768435.495853:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350930e00 x1862023076975872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495863:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076975872/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495871:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495875:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768435.495885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350932300 x1862023076975744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495896:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076975744/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495907:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495910:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975744:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768435.495917:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350931880 x1862023076975488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495928:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076975488/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495936:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495940:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495943:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975488:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768435.495950:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350932a00 x1862023076975616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495962:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076975616/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.495970:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.495973:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.495977:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076975616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768435.495984:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768432/real 1775768435] req@ffffa04350930700 x1862023076975360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768435.495997:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076975360/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768435.496004:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768435.496008:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768435.496011:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076975360:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768435.687177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768435.687177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768436.519172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768436.519204:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768436.711157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768436.711158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768437.543161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768437.543222:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543229:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768437.543233:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543245:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543257:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543264:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:3.0:1775768437.543292:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768437.543306:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076976512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543334:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543338:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768437.543341:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543346:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543351:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543355:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543368:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768437.543383:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076976640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543394:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543397:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768437.543399:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543404:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543408:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543412:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543417:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768437.543427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076976768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768437.543434:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076976512:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768437.543436:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543439:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768437.543440:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543443:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543447:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543451:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543455:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768437.543464:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076976896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543472:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543475:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768437.543476:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543480:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543483:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543486:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543490:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768437.543500:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076977024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543507:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543525:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768437.543527:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543531:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543539:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543548:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768437.543551:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543553:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768437.543563:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076977152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543572:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543576:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768437.543578:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543582:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543591:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543595:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543600:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768437.543611:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076977280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543620:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543624:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768437.543626:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543630:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543634:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543639:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543649:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768437.543662:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076977408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768437.543681:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768437.543685:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768437.543686:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768437.543690:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4939 00000100:00080000:0.0:1775768437.543694:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768437.543698:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768437.543703:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768437.543714:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076977536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768437.544145:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349640700 x1862023076976512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544169:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076976512/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544184:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544187:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076976512:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768437.544198:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976640:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768437.544236:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544250:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349641c00 x1862023076976640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076976640/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544275:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544279:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544283:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976640:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768437.544317:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976768:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768437.544338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544349:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349641880 x1862023076976768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544360:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076976768/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544368:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544371:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544374:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768437.544380:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976896:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768437.544400:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544409:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349643800 x1862023076976896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544419:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076976896/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544427:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544430:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544433:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076976896:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768437.544440:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977024:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768437.544470:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544481:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349643b80 x1862023076977024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544492:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076977024/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544501:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544503:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544506:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768437.544521:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768437.544541:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544549:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349641180 x1862023076977152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544561:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076977152/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544571:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544574:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544577:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768437.544582:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768437.544598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544607:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349640a80 x1862023076977280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544617:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076977280/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544624:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544628:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768437.544634:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768437.544657:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349643100 x1862023076977408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544687:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076977408/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544697:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544702:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768437.544710:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768437.544729:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768437.544739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768437/real 1775768437] req@ffffa04349642a00 x1862023076977536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768437.544750:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076977536/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768437.544757:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768437.544761:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768437.544764:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076977536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768437.735168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768437.735168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768438.567151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768438.567170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768438.759122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768438.759122:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768439.591091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768439.591130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768439.783140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768439.783141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768440.615148:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768440.615300:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768440.807112:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768440.807117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768441.639155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768441.639161:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768441.831130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768441.831130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768442.663156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768442.663189:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663198:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768442.663203:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663216:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663232:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775768442.663235:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768442.663241:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768442.663295:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076977664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663326:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663330:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768442.663331:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663336:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663340:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663344:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663357:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768442.663369:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076977792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663377:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663381:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768442.663395:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663399:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663402:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663405:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663411:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768442.663421:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076977920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663430:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663434:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768442.663435:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663439:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663442:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663445:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663450:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768442.663458:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076978048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663480:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663483:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768442.663485:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663489:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663492:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663496:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663501:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768442.663511:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076978176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663520:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663523:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768442.663525:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663529:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663532:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663536:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663540:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768442.663543:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076977664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768442.663549:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076978304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663558:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663561:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768442.663562:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663566:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663568:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663571:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663575:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768442.663583:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076978432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663591:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663594:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768442.663596:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663599:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4944 00000100:00080000:0.0:1775768442.663601:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663604:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663608:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768442.663616:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076978560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768442.663624:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768442.663627:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768442.663628:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768442.663632:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4944 00000800:00000100:3.0:1775768442.663633:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768442.663635:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768442.663638:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768442.663642:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768442.663650:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076978688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768442.663677:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349641500 x1862023076977664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.663704:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076977664/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.663713:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.663718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.663722:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076977664:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768442.663729:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977792:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768442.663752:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.663763:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349640380 x1862023076977792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.663774:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076977792/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.663783:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.663786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.663789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977792:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768442.663807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977920:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768442.663829:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.663840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349642680 x1862023076977920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.663851:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076977920/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.663857:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.663860:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.663863:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076977920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768442.663868:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978048:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768442.663894:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.663903:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349640e00 x1862023076978048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.663914:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076978048/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.663921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.663924:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.663928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978048:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768442.663934:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978176:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768442.663950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.663959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349642300 x1862023076978176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.663969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076978176/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.663977:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.663980:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.663983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768442.664017:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768442.664062:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.664072:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349642a00 x1862023076978304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.664080:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076978304/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.664086:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.664088:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.664089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768442.664093:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768442.664104:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.664110:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349643100 x1862023076978432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.664116:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076978432/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.664123:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.664125:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.664128:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768442.664133:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768442.664149:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.664159:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349640a80 x1862023076978560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.664171:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076978560/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.664178:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.664181:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.664184:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768442.664189:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768442.664202:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768442.664208:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768442/real 1775768442] req@ffffa04349641180 x1862023076978688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768442.664215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076978688/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768442.664220:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768442.664222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768442.664224:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076978688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768442.855212:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768442.855294:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768443.687165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768443.687221:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768443.879145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768443.879553:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768444.711203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768444.711386:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768444.903581:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768444.904655:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768445.735180:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768445.735210:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768445.927167:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768445.927174:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768446.759116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768446.759153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768446.951181:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768446.951184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768447.783154:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768447.783201:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768447.783286:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783291:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768447.783294:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783303:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783311:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783317:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768447.783352:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076978816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783378:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783381:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768447.783383:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783386:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783390:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783393:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783401:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768447.783427:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076978944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783436:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783440:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768447.783441:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783445:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783448:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783451:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783455:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768447.783464:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076979072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783472:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783475:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768447.783476:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783479:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783482:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783485:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783489:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768447.783497:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076979200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783504:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783507:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768447.783508:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783511:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783514:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783517:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783521:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768447.783529:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076979328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783537:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783540:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768447.783541:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783544:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783547:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783551:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783555:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768447.783565:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076979456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783572:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783574:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768447.783575:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783578:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783581:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783584:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783587:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768447.783593:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076979584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783601:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783604:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768447.783605:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783608:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783611:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783614:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783617:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768447.783625:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076979712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768447.783633:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768447.783635:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768447.783637:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768447.783639:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4949 00000100:00080000:0.0:1775768447.783642:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768447.783644:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768447.783648:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768447.783655:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076979840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768447.783717:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076978816:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768447.783771:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.783807:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349643b80 x1862023076978816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.783823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076978816/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.783833:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.783837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.783840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076978816:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768447.783847:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978944:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768447.783877:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.783888:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349643800 x1862023076978944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.783905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076978944/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.783913:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.783916:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.783919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076978944:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768447.783925:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979072:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768447.783941:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.783949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349641880 x1862023076979072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.783959:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076979072/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.783978:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.783982:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.783985:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979072:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768447.783990:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979200:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768447.784009:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784018:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349641c00 x1862023076979200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784051:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076979200/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979200:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768447.784075:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768447.784094:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784105:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349640700 x1862023076979328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076979328/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784125:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784128:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784131:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076979328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768447.784137:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768447.784154:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784163:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349641180 x1862023076979456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784174:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076979456/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784186:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768447.784195:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768447.784211:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784220:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349640a80 x1862023076979584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784232:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076979584/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784240:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784243:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784246:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768447.784252:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768447.784268:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784277:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349643100 x1862023076979712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784287:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076979712/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784294:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784297:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784300:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768447.784305:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768447.784322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768447.784333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768447/real 1775768447] req@ffffa04349642a00 x1862023076979840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768447.784344:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076979840/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768447.784352:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768447.784354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768447.784357:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076979840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768447.975204:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768447.975226:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768448.807133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768448.807164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768448.999178:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768448.999185:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768449.831143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768449.831286:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768450.023162:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768450.023165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768450.855138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768450.855183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768451.047169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768451.047463:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768451.879169:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768451.879416:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768452.071121:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768452.071121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768452.903146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768452.903169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768452.903239:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903245:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768452.903249:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903258:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903267:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903273:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768452.903305:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076979968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768452.903328:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903331:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768452.903333:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903336:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903339:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903343:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903352:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768452.903377:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076979968:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768452.903384:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076980096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768452.903395:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903398:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768452.903400:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903404:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903408:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903411:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903417:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768452.903427:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076980224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768452.903437:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768452.903439:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768452.903443:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768452.903445:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903449:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903453:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903457:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903461:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768452.903471:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076980352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768452.903480:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903483:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768452.903485:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903488:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903490:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903498:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903502:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768452.903512:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076980480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768452.903522:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903525:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768452.903527:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903531:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903535:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768452.903538:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7e680 x1862023076979968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768452.903541:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903549:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768452.903561:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076980608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768452.903566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076979968/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768452.903578:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903583:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768452.903585:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768452.903585:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.903594:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768452.903595:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903600:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768452.903602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076979968:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768452.903606:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903611:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768452.903627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768452.903632:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076980736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768452.903652:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768452.903663:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768452.903666:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7df80 x1862023076980096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768452.903669:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768452.903672:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903681:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903686:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768452.903690:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076980096/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768452.903692:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768452.903701:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768452.903708:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768452.903708:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.903715:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980096:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768452.903723:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076980864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768452.903724:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980224:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768452.903734:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768452.903738:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768452.903741:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768452.903747:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4954 00000100:00080000:2.0:1775768452.903751:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768452.903754:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768452.903759:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768452.903766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768452.903772:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076980992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768452.903781:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7ed80 x1862023076980224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.903805:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076980224/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.903813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.903816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.903819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768452.903825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980352:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768452.903845:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.903855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7d880 x1862023076980352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.903866:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076980352/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.903881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.903884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.903886:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980352:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768452.903890:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768452.903904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.903910:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7ea00 x1862023076980480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.903917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076980480/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.903922:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.903930:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.903933:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076980480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768452.903938:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768452.903955:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.903965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7f800 x1862023076980608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.903976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076980608/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.903984:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.903986:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.903990:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768452.904003:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768452.904017:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.904023:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7c700 x1862023076980736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.904051:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076980736/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.904058:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.904061:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.904063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768452.904069:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768452.904087:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.904097:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7ce00 x1862023076980864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.904105:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076980864/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.904110:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.904113:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.904116:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768452.904121:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768452.904138:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768452.904148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768452/real 1775768452] req@ffffa0434aa7fb80 x1862023076980992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768452.904159:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076980992/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768452.904167:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768452.904169:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768452.904172:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076980992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768453.095216:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768453.095652:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768453.927200:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768453.927324:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768454.119162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768454.119167:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768454.953243:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768454.953583:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768455.143196:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768455.143251:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768455.975241:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768455.975566:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768456.167108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768456.168396:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768456.999366:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768456.999417:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768457.191277:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768457.191284:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768458.023146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768458.023169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768458.023395:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023401:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768458.023404:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023414:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023423:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023428:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768458.023463:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076981120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768458.023487:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023491:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768458.023492:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023496:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023500:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023505:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023513:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768458.023523:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076981120:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768458.023528:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076981248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768458.023540:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023543:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768458.023544:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023547:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023550:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023553:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023557:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768458.023565:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076981376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768458.023572:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023574:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768458.023575:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023577:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023579:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023581:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768458.023586:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023588:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768458.023600:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076981504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768458.023609:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023612:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768458.023614:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023618:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023622:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768458.023627:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7fb80 x1862023076981120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768458.023628:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023635:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768458.023648:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076981632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768458.023654:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076981120/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768458.023656:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023662:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768458.023664:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023671:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023675:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768458.023675:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768458.023680:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768458.023683:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:2.0:1775768458.023685:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768458.023690:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076981120:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768458.023694:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076981760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768458.023701:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023703:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768458.023705:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023707:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023709:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023711:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023713:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768458.023720:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076981888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768458.023720:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981248:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768458.023732:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023734:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768458.023735:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768458.023739:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023742:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023745:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023748:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768458.023758:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076982016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768458.023766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768458.023774:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768458.023777:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768458.023779:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768458.023783:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7ce00 x1862023076981248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768458.023786:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4959 00000100:00080000:2.0:1775768458.023793:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768458.023800:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768458.023803:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768458.023805:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076981248/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768458.023814:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076982144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768458.023816:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.023825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.023830:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981248:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768458.023837:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981376:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768458.023855:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.023864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7c700 x1862023076981376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.023875:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076981376/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.023881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.023884:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.023887:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768458.023892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981504:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768458.023916:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.023926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7f800 x1862023076981504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.023936:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076981504/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.023944:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.023947:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.023950:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981504:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768458.023955:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768458.023970:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.023978:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7ea00 x1862023076981632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.023987:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076981632/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.023995:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.023997:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.024000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076981632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768458.024005:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076981760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768458.024019:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.024047:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7d880 x1862023076981760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.024058:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076981760/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.024066:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.024068:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.024072:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076981760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768458.024088:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076981888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768458.024105:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.024114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7ed80 x1862023076981888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.024124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076981888/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.024131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.024133:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.024136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076981888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768458.024141:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768458.024154:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.024162:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7df80 x1862023076982016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.024172:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076982016/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.024179:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.024182:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.024185:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768458.024190:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768458.024203:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768458.024211:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768458/real 1775768458] req@ffffa0434aa7e680 x1862023076982144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768458.024221:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076982144/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768458.024228:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768458.024231:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768458.024233:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768458.215128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768458.215128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768459.047135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768459.047168:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768459.239152:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768459.239152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768460.071400:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768460.071741:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768460.263130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768460.263130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768461.095894:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768461.097296:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768461.287161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768461.288398:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768462.119768:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768462.119877:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768462.311152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768462.311158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775768463.143113:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143123:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768463.143127:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143138:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143147:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768463.143223:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768463.143306:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768463.143307:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768463.143346:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076982272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768463.143407:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076982272:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768463.143454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768463.143484:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143486:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768463.143488:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143491:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143494:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143496:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143506:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768463.143519:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076982400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768463.143571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350930700 x1862023076982272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.143584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076982272/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.143590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.143593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.143595:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076982272:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768463.143614:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982400:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768463.143637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.143647:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350932a00 x1862023076982400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.143658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076982400/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.143666:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.143668:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.143671:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982400:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768463.143718:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143721:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768463.143724:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143728:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143732:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143741:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143747:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768463.143759:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076982528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143768:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143771:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768463.143772:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143776:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143779:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143782:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143786:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768463.143793:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076982656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143801:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143804:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768463.143805:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143808:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143811:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143814:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143824:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768463.143839:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076982784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143847:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143850:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768463.143851:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143855:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143858:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143861:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143865:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768463.143873:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076982912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143880:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143882:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768463.143884:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143886:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143889:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143892:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143895:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768463.143902:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076983040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143916:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143919:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768463.143920:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143923:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143926:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143929:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143933:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768463.143941:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076983168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768463.143948:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768463.143951:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768463.143953:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768463.143956:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4964 00000100:00080000:1.0:1775768463.143959:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768463.143962:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768463.143966:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768463.143975:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076983296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768463.144045:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982528:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768463.144067:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144078:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350931880 x1862023076982528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076982528/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768463.144108:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982656:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768463.144125:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144134:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350932300 x1862023076982656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144144:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076982656/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144155:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144158:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144160:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982656:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768463.144164:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768463.144181:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144190:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350930e00 x1862023076982784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144200:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076982784/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144208:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144210:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144213:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076982784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768463.144219:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768463.144236:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144246:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350932680 x1862023076982912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144254:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076982912/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144258:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144260:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144262:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076982912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768463.144277:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768463.144291:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144297:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350931c00 x1862023076983040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144307:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076983040/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144314:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144317:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144319:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768463.144324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768463.144342:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144351:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350930a80 x1862023076983168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076983168/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144363:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144366:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768463.144369:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768463.144380:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768463.144386:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768463/real 1775768463] req@ffffa04350930000 x1862023076983296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768463.144392:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076983296/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768463.144396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768463.144397:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768463.144399:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076983296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768463.335142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768463.335296:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768464.167144:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768464.168357:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768464.359094:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768464.359100:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768465.191180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768465.192536:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768465.384084:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768465.384183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768466.215145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768466.215187:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768466.407369:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768466.408439:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768467.239145:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768467.239298:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768467.431149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768467.432307:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768468.263194:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768468.264698:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.264707:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768468.264713:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.264724:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.264736:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.264743:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768468.264782:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076983424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.264813:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.264818:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768468.264820:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.264825:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.264829:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.264835:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.264848:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768468.264860:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076983424:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768468.264863:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076983552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.264877:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.264881:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768468.264883:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.264887:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.264891:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768468.264896:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768468.264897:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.264906:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768468.264919:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076983680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.264930:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.264933:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768468.264937:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775768468.264937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768468.264944:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.264948:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.264952:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.264958:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768468.264970:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076983808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.264979:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.264985:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768468.264986:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7c000 x1862023076983424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768468.264987:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.264994:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.265001:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.265006:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.265011:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768468.265013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076983424/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768468.265023:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076983936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768468.265045:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265051:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076983424:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768468.265065:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.265069:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768468.265071:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.265076:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.265081:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.265085:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265090:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983552:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768468.265092:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768468.265106:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076984064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.265117:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768468.265118:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768468.265123:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768468.265126:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.265131:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.265134:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.265138:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.265143:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768468.265152:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076984192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.265160:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.265163:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768468.265165:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.265168:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.265171:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.265174:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.265178:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768468.265187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076984320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768468.265196:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768468.265199:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768468.265200:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768468.265204:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4969 00000100:00080000:2.0:1775768468.265207:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768468.265211:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768468.265227:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768468.265239:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076984448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768468.265243:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7e300 x1862023076983552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076983552/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265277:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265281:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265285:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983552:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768468.265293:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983680:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768468.265311:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265319:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7c380 x1862023076983680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265330:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076983680/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265338:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265341:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265344:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983680:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768468.265348:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983808:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768468.265361:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265368:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7f480 x1862023076983808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265378:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076983808/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265386:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265389:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265392:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983808:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768468.265396:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768468.265408:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265415:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7f100 x1862023076983936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265424:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076983936/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265432:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265438:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076983936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768468.265456:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768468.265471:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265479:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7dc00 x1862023076984064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265489:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076984064/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265497:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265500:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265503:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768468.265508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768468.265520:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265527:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7d500 x1862023076984192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265537:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076984192/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265544:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768468.265554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768468.265566:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265573:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7c000 x1862023076984320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265584:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076984320/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265590:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265596:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768468.265600:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768468.265611:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768468.265618:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768468/real 1775768468] req@ffffa0434aa7e680 x1862023076984448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768468.265628:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076984448/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768468.265635:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768468.265637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768468.265640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076984448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768468.455143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768468.455686:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768469.287160:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768469.287610:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768469.479331:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768469.479350:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768470.311169:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768470.311252:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768470.503158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768470.503166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768471.335160:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768471.335210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768471.527140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768471.527246:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768472.359146:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768472.359207:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768472.551130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768472.551131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768473.383132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768473.383158:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383179:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775768473.383180:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768473.383183:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383193:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383211:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383218:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768473.383248:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076984576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383279:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383283:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768473.383285:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383289:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383292:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383296:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383305:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768473.383317:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076984704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383325:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383328:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768473.383330:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383333:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383336:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383339:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768473.383351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076984832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383358:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383360:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768473.383362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383376:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383379:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383383:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383387:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768473.383397:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076984960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383406:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768473.383411:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383414:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383417:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383421:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383425:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768473.383434:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076985088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383442:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383445:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768473.383446:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383450:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383454:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383457:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383462:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768473.383471:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076985216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383479:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383482:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768473.383484:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383487:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383489:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383493:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383496:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768473.383504:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076985344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383512:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383515:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768473.383516:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.383519:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4975 00000100:00080000:2.0:1775768473.383522:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.383525:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768473.383529:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768473.383538:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076985472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768473.383545:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768473.383548:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768473.383589:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076984576:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768473.383992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384055:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7df80 x1862023076984576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384075:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076984576/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384085:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384090:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384094:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076984576:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768473.384114:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984704:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768473.384136:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7ed80 x1862023076984704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384188:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076984704/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384195:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384198:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384200:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984704:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768473.384206:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984832:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768473.384224:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384233:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7d880 x1862023076984832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384243:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076984832/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384250:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384252:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384255:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984832:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768473.384259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984960:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768473.384275:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384283:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7ea00 x1862023076984960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384309:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076984960/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384317:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384320:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384323:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076984960:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768473.384329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768473.384348:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7f800 x1862023076985088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384369:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076985088/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384376:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384379:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384382:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768473.384387:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768473.384404:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384414:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7c700 x1862023076985216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384425:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076985216/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384438:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384441:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384444:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768473.384449:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768473.384467:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384476:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7ce00 x1862023076985344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076985344/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384494:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384497:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384500:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768473.384506:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985472:192.168.202.158@tcp:8:kworker.0 00000100:00080000:2.0:1775768473.384517:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768473.384524:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4975 00000800:00000100:3.0:1775768473.384527:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768473.384532:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768473.384538:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7fb80 x1862023076985472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768473.384546:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768473.384562:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076985600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768473.384575:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076985472/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384587:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384591:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768473.384631:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768473.384651:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768473.384661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768473/real 1775768473] req@ffffa0434aa7e680 x1862023076985600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768473.384672:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076985600/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768473.384680:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768473.384683:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768473.384685:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076985600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768473.575179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768473.577560:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768474.407274:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768474.408778:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768474.599136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768474.599181:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768475.431173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768475.431217:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768475.623124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768475.623211:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768476.455157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768476.455244:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768476.647157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768476.647157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768477.479102:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768477.479233:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768477.671131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768477.671161:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768478.503200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768478.503250:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768478.503284:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.503291:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768478.503295:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.503329:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.503347:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768478.503363:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768478.503448:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076985728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768478.503550:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.503554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775768478.503568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076985728:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768478.503568:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.503575:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.503580:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768478.503583:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768478.503608:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768478.503623:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076985856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768478.503628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768478.503639:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.503642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768478.503644:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.503649:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00100000:1.0:1775768478.503668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825d880 x1862023076985728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.503671:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768478.503686:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076985728/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.503696:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.503699:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.503704:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076985728:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768478.503723:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985856:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768478.503741:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768478.503752:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825dc00 x1862023076985856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768478.503763:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076985856/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.503771:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.503774:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.503778:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985856:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768478.504073:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768478.504079:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768478.504091:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076985984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768478.504137:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.504141:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768478.504143:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.504153:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.504157:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768478.504161:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768478.504166:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768478.504189:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076986112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768478.504200:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768478.504201:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985984:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768478.504211:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768478.504213:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.504218:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.504231:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768478.504235:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:1.0:1775768478.504241:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:3.0:1775768478.504248:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768478.504255:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825d500 x1862023076985984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768478.504270:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076985984/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.504279:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768478.504280:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076986240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768478.504284:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.504288:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076985984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768478.504300:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076986112:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768478.504322:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.504326:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768478.504328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768478.504332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768478.504342:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00100000:1.0:1775768478.504346:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825e300 x1862023076986112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504358:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768478.504362:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076986112/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.504371:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768478.504374:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768478.504377:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000008:00100000:3.0:1775768478.504381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768478.504382:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076986112:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768478.504394:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076986368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768478.504409:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076986240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768478.504435:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768478.504446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768478.504448:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825e680 x1862023076986240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504451:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768478.504463:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076986240/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504464:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768478.504477:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:1.0:1775768478.504482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.504487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775768478.504490:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768478.504491:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076986240:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768478.504497:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768478.504503:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768478.504508:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768478.504516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076986496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768478.504527:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.504530:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775768478.504534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768478.504534:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768478.504546:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825f100 x1862023076986368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504555:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.504561:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768478.504567:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076986368/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504567:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768478.504579:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.504583:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.504587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986368:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768478.504587:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768478.504603:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076986624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768478.504605:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768478.504622:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768478.504625:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768478.504629:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768478.504630:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768478.504631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825ea00 x1862023076986496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504639:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4980 00000100:00080000:3.0:1775768478.504646:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768478.504650:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076986496/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768478.504652:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768478.504660:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.504662:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.504664:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986496:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768478.504671:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768478.504674:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768478.504686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768478.504692:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825ed80 x1862023076986624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768478.504700:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076986752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768478.504702:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076986624/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.504717:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.504720:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.504724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768478.504740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768478.504760:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768478.504770:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768478/real 1775768478] req@ffffa0435825c000 x1862023076986752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768478.504780:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076986752/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768478.504788:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768478.504790:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768478.504793:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076986752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768478.695191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768478.695229:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768479.527210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768479.527483:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768479.719139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768479.719140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768480.551168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768480.551171:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768480.743153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768480.743153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768481.575132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768481.576214:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768481.767148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768481.767151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768482.599558:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768482.599651:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768482.791156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768482.791163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768483.623110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768483.623144:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768483.623151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623158:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768483.623163:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623172:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623181:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623188:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768483.623220:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076986880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623250:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623253:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768483.623255:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623259:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623263:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623274:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768483.623284:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076987008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623296:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768483.623298:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623302:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623305:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623308:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623311:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768483.623319:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076987136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623330:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768483.623332:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623335:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623338:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623341:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623344:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768483.623351:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076987264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623359:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623361:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768483.623363:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623366:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623371:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623375:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768483.623381:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076987392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623388:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623391:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768483.623392:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623395:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623398:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623401:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623404:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768483.623411:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076987520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623418:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623421:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768483.623423:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623426:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623429:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623432:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623436:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768483.623443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076987648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623450:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623453:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768483.623454:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623457:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623459:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623462:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623466:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768483.623474:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076987776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768483.623482:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768483.623484:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768483.623486:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768483.623489:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4985 00000100:00080000:2.0:1775768483.623492:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768483.623494:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768483.623497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768483.623504:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076987904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768483.623568:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076986880:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775768483.623619:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623648:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7c000 x1862023076986880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623664:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076986880/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623674:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623678:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623681:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076986880:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768483.623688:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987008:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768483.623704:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623714:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7d500 x1862023076987008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623725:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076987008/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623734:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623737:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987008:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768483.623758:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987136:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768483.623773:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7dc00 x1862023076987136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623793:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076987136/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623800:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623802:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623805:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768483.623811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987264:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768483.623825:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7f100 x1862023076987264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076987264/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987264:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768483.623861:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768483.623874:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7f480 x1862023076987392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076987392/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076987392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768483.623909:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987520:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768483.623925:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623934:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7c380 x1862023076987520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.623945:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076987520/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.623963:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.623966:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.623970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768483.623975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768483.623990:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.623998:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7e300 x1862023076987648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.624009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076987648/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.624016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.624019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.624022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768483.624042:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987776:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768483.624058:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.624079:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7e680 x1862023076987776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.624090:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076987776/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.624098:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.624101:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.624103:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768483.624108:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768483.624123:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768483.624131:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768483/real 1775768483] req@ffffa0434aa7fb80 x1862023076987904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768483.624141:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076987904/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768483.624148:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768483.624151:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768483.624154:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076987904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768483.815174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768483.815744:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768484.647141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768484.647175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768484.839151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768484.839152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768485.671114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768485.671190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768485.863127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768485.864116:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768486.695129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768486.695165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768486.887145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768486.887173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768487.719142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768487.719277:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768487.911109:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768487.911149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768488.743125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768488.743148:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743153:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768488.743157:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743165:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743172:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743177:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768488.743190:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768488.743206:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076988032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743225:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743227:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768488.743228:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743231:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743235:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743239:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743248:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768488.743254:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076988032:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768488.743262:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023076988160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743273:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743277:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768488.743279:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743282:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743287:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743290:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743295:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768488.743302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768488.743305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023076988288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743312:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768488.743316:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743319:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743321:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743324:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743327:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768488.743333:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7ce00 x1862023076988032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768488.743336:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023076988416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743359:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768488.743361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768488.743364:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076988032/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768488.743364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743373:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743379:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1775768488.743386:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768488.743387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023076988544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768488.743395:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743403:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076988032:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768488.743405:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743410:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768488.743412:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743416:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743418:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743420:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743422:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768488.743427:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076988672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768488.743433:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988160:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768488.743436:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743442:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768488.743445:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743448:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:2.0:1775768488.743449:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743451:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743454:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768488.743468:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768488.743469:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076988800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743476:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743478:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768488.743479:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743482:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00100000:1.0:1775768488.743485:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7c700 x1862023076988160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768488.743486:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743492:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768488.743497:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768488.743510:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023076988160/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768488.743510:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076988928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768488.743532:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768488.743535:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768488.743537:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768488.743537:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768488.743543:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4990 00000100:00080000:1.0:1775768488.743546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768488.743547:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768488.743553:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743554:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988160:192.168.202.158@tcp:38:kworker.0 00000008:00100000:2.0:1775768488.743556:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768488.743566:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768488.743567:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076989056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768488.743590:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743601:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7f800 x1862023076988288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743610:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023076988288/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743615:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743617:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743619:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768488.743622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988416:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768488.743635:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743641:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7ea00 x1862023076988416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743651:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023076988416/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743662:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743664:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743666:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988416:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768488.743670:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988544:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768488.743679:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743685:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7d880 x1862023076988544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743691:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023076988544/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743695:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743697:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743699:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076988544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768488.743709:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768488.743718:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7ed80 x1862023076988672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076988672/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743736:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743738:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743740:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768488.743742:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768488.743750:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743754:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7df80 x1862023076988800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076988800/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743767:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768488.743770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768488.743777:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743782:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7ce00 x1862023076988928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743788:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076988928/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743793:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743795:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076988928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768488.743798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768488.743806:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768488.743810:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768488/real 1775768488] req@ffffa0434aa7fb80 x1862023076989056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768488.743816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076989056/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768488.743820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768488.743822:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768488.743823:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768488.935123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768488.935161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768489.767115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768489.767162:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768489.959127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768489.959135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768490.791118:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768490.791192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768490.983125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768490.983125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768491.815125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768491.815177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768492.007126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768492.007126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768492.839129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768492.839196:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768493.031111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768493.031111:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768493.863091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768493.863120:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863126:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768493.863131:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863140:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 4995 00010000:00010000:2.0:1775768493.863147:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768493.863149:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863156:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768493.863187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076989184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768493.863208:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863210:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768493.863211:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863214:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863216:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863219:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863227:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768493.863233:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076989184:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768493.863236:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076989312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768493.863244:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863246:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768493.863246:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863249:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863250:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863252:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863255:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768493.863261:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076989440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768493.863267:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863269:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768493.863270:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863272:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863275:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863277:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863280:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768493.863282:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768493.863288:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076989568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768493.863294:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863296:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768493.863297:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863299:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863301:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863303:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863306:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768493.863316:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076989696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768493.863317:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349642300 x1862023076989184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768493.863340:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863342:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768493.863343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863345:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863347:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863349:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863352:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768493.863356:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076989184/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768493.863360:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076989824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768493.863367:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863375:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768493.863380:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768493.863382:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076989184:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768493.863383:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768493.863386:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863390:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863391:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768493.863394:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863396:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768493.863402:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076989952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768493.863404:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989312:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768493.863416:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768493.863419:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768493.863420:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863423:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863425:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768493.863428:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768493.863431:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863435:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768493.863441:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349640e00 x1862023076989312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768493.863444:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076990080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768493.863459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768493.863461:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076989312/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768493.863462:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768493.863465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768493.863471:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 4995 00000100:00080000:0.0:1775768493.863476:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768493.863476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768493.863479:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768493.863482:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768493.863482:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863486:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989312:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768493.863490:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076990208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768493.863493:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989440:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768493.863512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863520:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349642680 x1862023076989440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863527:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076989440/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863531:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863533:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863535:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768493.863538:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989568:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768493.863546:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863551:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349640380 x1862023076989568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863557:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076989568/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863562:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863563:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863565:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989568:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768493.863567:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989696:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768493.863575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863580:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349641500 x1862023076989696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076989696/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863591:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863594:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076989696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768493.863596:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768493.863604:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863608:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349642a00 x1862023076989824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863614:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076989824/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863619:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863621:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863622:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768493.863634:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989952:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768493.863642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863646:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349643100 x1862023076989952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863652:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076989952/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863656:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863660:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076989952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768493.863662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768493.863670:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349640a80 x1862023076990080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076990080/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863689:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863691:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863693:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768493.863696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990208:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768493.863704:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768493.863709:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768493/real 1775768493] req@ffffa04349641180 x1862023076990208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768493.863715:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076990208/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768493.863719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768493.863721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768493.863722:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768494.055131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768494.055131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768494.887200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768494.887234:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768495.079104:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768495.079140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768495.911295:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768495.911506:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768496.103130:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768496.103131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768496.935216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768496.935351:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768497.127128:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768497.127128:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768497.959121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768497.959175:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768498.151137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768498.151137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768498.983134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768498.983235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768498.983254:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983259:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768498.983262:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983271:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983281:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983287:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768498.983321:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076990336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768498.983346:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983350:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768498.983351:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983355:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983360:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983365:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983374:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768498.983393:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076990336:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768498.983406:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076990464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768498.983417:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983421:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768498.983423:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983427:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983431:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983435:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983440:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768498.983452:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076990592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768498.983462:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983465:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768498.983467:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983471:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983475:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983479:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983485:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768498.983495:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076990720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768498.983500:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990464:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768498.983513:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983517:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768498.983519:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983524:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983528:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983532:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983538:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768498.983541:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768498.983552:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076990848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768498.983561:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983565:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768498.983567:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983571:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00100000:1.0:1775768498.983574:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990720:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768498.983577:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983584:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983589:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768498.983602:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076990976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768498.983609:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990848:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768498.983615:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983618:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768498.983621:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983626:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983631:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768498.983636:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990976:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768498.983637:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983646:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768498.983659:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076991104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768498.983675:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983678:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768498.983680:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983682:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983685:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983688:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983692:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768498.983699:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076991232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768498.983708:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768498.983717:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768498.983720:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768498.983722:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768498.983726:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5000 00000100:00080000:3.0:1775768498.983730:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768498.983734:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768498.983739:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768498.983741:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768498.983752:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023076991360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768498.983768:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768499.175131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768499.175131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768500.007193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768500.007206:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768500.199135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768500.199135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768501.031188:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768501.031200:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768501.223092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768501.223151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768502.055215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768502.055336:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:3.0:1775768502.055553:0:2397:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:3.0:1775768502.055602:0:2397:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775768502.055617:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055666:0:2397:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055680:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055689:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055698:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055707:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055714:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055720:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:3.0:1775768502.055727:0:2397:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775768502.055923:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825dc00 x1862023076991360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.055962:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023076991360/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.055974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.055979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.055983:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991360:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768502.055990:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825d500 x1862023076991232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056001:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076991232/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056008:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056011:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056014:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991232:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768502.056021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825e300 x1862023076991104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076991104/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056509:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056513:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056516:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076991104:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768502.056523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825e680 x1862023076990976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056535:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076990976/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056543:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056546:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076990976:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768502.056556:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825ea00 x1862023076990720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056566:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076990720/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056573:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056576:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056579:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990720:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768502.056584:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825f100 x1862023076990848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056593:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076990848/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056600:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056604:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990848:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768502.056613:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825c000 x1862023076990464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056625:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076990464/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056633:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990464:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768502.056647:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825ed80 x1862023076990592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056656:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076990592/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056662:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056665:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056668:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076990592:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768502.056674:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768498/real 1775768502] req@ffffa0435825ce00 x1862023076990336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768502.056686:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076990336/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768502.056694:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768502.056697:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768502.056700:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076990336:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768502.247153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768502.247159:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768503.079162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768503.079286:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768503.271153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768503.271153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775768504.103156:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103164:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768504.103168:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103177:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103185:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103191:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768504.103193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768504.103207:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768504.103220:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076991488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768504.103241:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103244:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768504.103245:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103248:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103250:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103254:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103262:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768504.103272:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076991616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768504.103280:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076991488:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768504.103286:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103291:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768504.103293:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103310:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103321:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103328:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103334:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768504.103345:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076991744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768504.103355:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103360:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775768504.103363:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768504.103364:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103372:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103377:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103382:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103389:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768504.103402:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076991872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768504.103412:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103416:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768504.103418:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768504.103426:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350930000 x1862023076991488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768504.103430:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103437:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103445:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103451:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768504.103456:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076991488/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768504.103467:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076992000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768504.103469:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768504.103479:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768504.103479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768504.103486:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1775768504.103488:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076991488:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768504.103489:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103500:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103505:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768504.103510:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103519:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768504.103520:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768504.103532:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076992128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768504.103551:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103554:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768504.103556:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103561:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103567:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768504.103572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768504.103574:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103582:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768504.103593:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350930a80 x1862023076991616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768504.103597:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076992256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768504.103630:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103634:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768504.103635:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076991616/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768504.103637:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103646:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103653:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768504.103658:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768504.103659:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103667:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768504.103667:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.103674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991616:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768504.103684:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076992384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768504.103685:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991744:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768504.103700:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768504.103703:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768504.103706:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768504.103711:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5005 00000100:00080000:1.0:1775768504.103715:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768504.103720:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768504.103724:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768504.103730:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768504.103739:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350931c00 x1862023076991744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768504.103745:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076992512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768504.103782:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076991744/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.103792:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.103796:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.103799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768504.103806:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991872:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768504.103827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.103840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350932680 x1862023076991872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.103852:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076991872/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.103876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.103879:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.103883:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076991872:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768504.103889:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768504.103911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.103922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350930e00 x1862023076992000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.103934:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076992000/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.103950:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.103954:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.103957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768504.103983:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992128:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768504.104005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.104016:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350932300 x1862023076992128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.104054:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076992128/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.104064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.104068:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.104072:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992128:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768504.104080:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768504.104104:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.104114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350931880 x1862023076992256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.104126:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076992256/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.104135:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.104138:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.104142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768504.104148:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992384:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768504.104169:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.104180:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350932a00 x1862023076992384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.104195:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076992384/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.104202:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.104205:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.104209:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992384:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768504.104214:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768504.104232:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768504.104242:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768504/real 1775768504] req@ffffa04350930700 x1862023076992512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768504.104255:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076992512/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768504.104263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768504.104266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768504.104269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076992512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768504.295138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768504.295138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768505.127184:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768505.127205:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768505.319095:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768505.319097:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768506.151150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768506.151224:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768506.343155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768506.343155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768507.175258:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768507.176578:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768507.367203:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768507.367211:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768508.199196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768508.199909:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768508.391184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768508.391184:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:3.0:1775768509.223136:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768509.223138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768509.223147:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768509.223151:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223161:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223170:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223177:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768509.223213:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023076992640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768509.223240:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223243:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768509.223245:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223249:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223253:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223257:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768509.223267:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768509.223270:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076992640:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768509.223282:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023076992768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768509.223293:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223296:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768509.223298:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223302:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223306:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223309:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768509.223314:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768509.223327:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768509.223327:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023076992896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768509.223337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768509.223343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223346:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223350:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223354:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768509.223359:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768509.223368:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825d880 x1862023076992640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768509.223372:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023076993024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768509.223392:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223395:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775768509.223398:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023076992640/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768509.223399:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223408:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223414:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768509.223415:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768509.223419:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768509.223423:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:3.0:1775768509.223427:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768509.223428:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076992640:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768509.223445:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023076993152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768509.223451:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992768:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768509.223463:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223467:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768509.223469:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223475:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5010 00000800:00000100:1.0:1775768509.223476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768509.223482:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223487:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223489:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825ce00 x1862023076992768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768509.223495:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768509.223500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023076992768/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223507:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768509.223510:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023076993280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768509.223512:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223517:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992768:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768509.223525:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992896:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768509.223531:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223534:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768509.223536:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768509.223540:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768509.223542:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:3.0:1775768509.223547:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768509.223548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825ed80 x1862023076992896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768509.223555:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223560:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023076992896/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768509.223562:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768509.223570:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223572:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223574:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076992896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768509.223577:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993024:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768509.223585:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023076993408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768509.223591:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223599:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825c000 x1862023076993024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768509.223599:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223607:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768509.223611:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768509.223614:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023076993024/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768509.223618:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5010 00000100:00080000:1.0:1775768509.223626:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768509.223627:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768509.223630:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223635:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993024:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768509.223635:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768509.223644:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768509.223657:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023076993536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768509.223662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993152:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768509.223673:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768509.223676:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768509.223678:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768509.223683:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5010 00000800:00000100:1.0:1775768509.223687:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768509.223688:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768509.223693:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768509.223698:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768509.223700:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825f100 x1862023076993152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768509.223712:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023076993664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768509.223716:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023076993152/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223736:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223741:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223743:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768509.223748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993280:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768509.223764:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223772:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825ea00 x1862023076993280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768509.223779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023076993280/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223785:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223787:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223789:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993280:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768509.223793:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768509.223821:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00010000:00010000:2.0:1775768509.223828:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768509.223832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825e680 x1862023076993408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768509.223840:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023076993408/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223845:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223847:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223849:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768509.223864:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993536:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768509.223875:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825e300 x1862023076993536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768509.223887:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023076993536/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223892:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223893:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223895:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993536:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768509.223899:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768509.223907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768509.223911:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768509/real 1775768509] req@ffffa0435825d500 x1862023076993664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768509.223916:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023076993664/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768509.223921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768509.223922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768509.223924:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076993664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768509.415130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768509.415131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768510.247141:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768510.247222:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768510.439125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768510.439125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768511.271097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768511.271193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768511.463092:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768511.463092:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768512.295114:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768512.295174:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768512.487132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768512.487132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768513.319110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768513.319129:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768513.511143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768513.511143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768514.343132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768514.343159:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343164:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768514.343168:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343177:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5016 00010000:00010000:0.0:1775768514.343183:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768514.343186:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343192:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768514.343229:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076993792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768514.343255:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343258:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768514.343260:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343264:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343268:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343272:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343281:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768514.343283:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076993792:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768514.343294:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076993920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768514.343304:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343308:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768514.343309:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343313:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343317:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343321:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343326:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768514.343332:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768514.343339:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076994048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768514.343349:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343352:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768514.343354:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343358:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343362:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343366:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343371:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768514.343372:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350930700 x1862023076993792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768514.343386:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076994176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768514.343390:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076993792/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343411:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768514.343412:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768514.343417:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768514.343418:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768514.343422:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768514.343422:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076993792:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768514.343432:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343435:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343439:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343444:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768514.343444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993920:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768514.343459:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076994304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768514.343471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768514.343471:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768514.343476:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768514.343478:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343482:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343486:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768514.343487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350932a00 x1862023076993920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768514.343495:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343503:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768514.343507:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076993920/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768514.343516:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076994432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768514.343517:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343525:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343533:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076993920:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768514.343534:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343540:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768514.343543:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768514.343544:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343552:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343557:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343561:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768514.343566:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768514.343575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768514.343579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076994560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768514.343588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350931880 x1862023076994048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768514.343589:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343593:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768514.343594:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343601:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343607:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768514.343612:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076994048/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768514.343618:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768514.343629:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768514.343633:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076994688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768514.343636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343642:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768514.343652:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994176:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768514.343655:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768514.343658:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768514.343660:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768514.343665:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5016 00000100:00080000:1.0:1775768514.343669:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768514.343671:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768514.343677:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350932300 x1862023076994176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768514.343683:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768514.343694:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076994176/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768514.343700:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076994816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768514.343702:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343708:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343714:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994176:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768514.343731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768514.343765:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343775:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350930e00 x1862023076994304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768514.343785:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076994304/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343790:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343792:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343794:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076994304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768514.343798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994432:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768514.343810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350932680 x1862023076994432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768514.343826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076994432/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343830:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343832:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343833:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994432:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768514.343836:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768514.343850:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343858:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350931c00 x1862023076994560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768514.343868:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076994560/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343876:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343878:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343881:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768514.343886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994688:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768514.343904:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343913:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350930a80 x1862023076994688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768514.343920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076994688/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343929:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343931:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343932:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994688:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768514.343935:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768514.343946:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768514.343951:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768514/real 1775768514] req@ffffa04350930000 x1862023076994816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768514.343957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076994816/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768514.343962:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768514.343963:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768514.343964:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076994816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768514.535117:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768514.535117:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768515.367091:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768515.367130:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768515.559120:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768515.559120:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768516.391140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768516.391202:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768516.583107:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768516.583108:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768517.415097:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768517.415165:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768517.607132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768517.607132:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768518.439157:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768518.439179:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768518.631098:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768518.631105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768519.463131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768519.463137:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463144:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768519.463148:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463158:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5021 00010000:00010000:3.0:1775768519.463160:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768519.463169:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463176:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768519.463212:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023076994944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768519.463240:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463243:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768519.463245:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463248:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463252:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463256:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463265:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768519.463277:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023076995072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768519.463288:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463293:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768519.463294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076994944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768519.463295:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463303:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463307:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463310:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463315:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768519.463325:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023076995200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768519.463335:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463338:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768519.463340:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463344:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463348:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463352:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463357:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768519.463366:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768519.463370:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023076995328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768519.463380:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463383:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768519.463385:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463389:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463392:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463396:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463400:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768519.463409:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349640700 x1862023076994944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768519.463414:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023076995456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768519.463438:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463442:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768519.463444:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463448:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00100000:3.0:1775768519.463453:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023076994944/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768519.463454:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768519.463461:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463469:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768519.463471:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463478:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768519.463481:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076995584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768519.463486:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076994944:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768519.463501:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463504:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768519.463506:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463511:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463515:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768519.463516:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768519.463521:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463527:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768519.463539:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076995712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768519.463548:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768519.463550:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768519.463554:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768519.463556:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768519.463560:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463564:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768519.463566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349641c00 x1862023076995072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768519.463569:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463577:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768519.463586:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023076995072/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768519.463590:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076995840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768519.463603:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768519.463606:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768519.463610:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768519.463611:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768519.463614:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768519.463616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995072:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768519.463621:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5021 00000100:00080000:0.0:1775768519.463626:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768519.463627:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995200:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768519.463632:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768519.463650:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1775768519.463650:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.463664:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349641880 x1862023076995200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768519.463666:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076995968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768519.463695:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023076995200/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.463723:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463726:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.463731:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768519.463738:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995328:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768519.463765:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.463778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349643800 x1862023076995328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.463790:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023076995328/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.463798:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463801:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.463804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995328:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768519.463809:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768519.463827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.463836:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349643b80 x1862023076995456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.463847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023076995456/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.463855:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.463861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076995456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768519.463866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995584:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768519.463885:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.463895:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349641180 x1862023076995584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.463905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076995584/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.463912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.463918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995584:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768519.463937:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768519.463956:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.463966:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349640a80 x1862023076995712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.463976:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076995712/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.463983:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.463987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.463989:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768519.463995:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995840:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768519.464013:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.464023:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349643100 x1862023076995840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.464063:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076995840/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.464071:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.464074:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.464077:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995840:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768519.464082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768519.464100:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768519.464111:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768519/real 1775768519] req@ffffa04349642a00 x1862023076995968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768519.464122:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076995968/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768519.464129:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768519.464133:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768519.464136:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076995968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768519.655142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768519.655142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768520.487107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768520.487166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768520.679143:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768520.679144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768521.511133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768521.511156:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768521.703124:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768521.703124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768522.535095:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768522.535164:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768522.727114:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768522.727114:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768523.559133:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768523.559155:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768523.751160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768523.751160:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:1.0:1775768524.583160:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583169:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768524.583174:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583185:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583193:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583200:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768524.583221:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768524.583245:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023076996096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768524.583271:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583274:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768524.583276:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583280:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583283:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583287:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583295:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768524.583307:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023076996224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768524.583317:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583321:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768524.583323:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583330:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00100000:3.0:1775768524.583332:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076996096:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768524.583334:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775768524.583334:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768524.583341:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583347:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768524.583359:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023076996352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768524.583369:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583372:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768524.583374:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583377:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583381:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583384:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583388:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768524.583398:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023076996480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768524.583404:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768524.583423:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583426:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768524.583428:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583432:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583437:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583441:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768524.583445:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350930000 x1862023076996096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768524.583447:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768524.583462:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023076996608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768524.583469:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023076996096/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768524.583474:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583481:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768524.583484:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583490:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:3.0:1775768524.583490:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768524.583495:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768524.583498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768524.583502:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768524.583505:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076996096:192.168.202.158@tcp:250:kworker.0 00000008:00100000:1.0:1775768524.583508:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768524.583515:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768524.583521:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023076996736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768524.583528:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583535:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768524.583536:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583539:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583542:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583546:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583550:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775768524.583559:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768524.583562:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023076996864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768524.583570:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583575:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775768524.583575:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350930a80 x1862023076996224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768524.583577:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768524.583582:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583584:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583587:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583591:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768524.583604:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023076996992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768524.583610:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023076996224/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768524.583616:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768524.583624:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768524.583627:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768524.583627:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.583632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:1.0:1775768524.583635:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5026 00000100:00080000:1.0:1775768524.583640:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768524.583643:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768524.583647:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768524.583665:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023076997120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768524.583814:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996224:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768524.583838:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768524.583870:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.583881:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350931c00 x1862023076996352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.583899:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023076996352/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.583907:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.583910:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.583913:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768524.583918:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996480:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768524.583936:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.583946:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350932680 x1862023076996480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.583957:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023076996480/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.583964:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.583967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.583970:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996480:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768524.583975:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768524.583992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.584001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350930e00 x1862023076996608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.584010:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023076996608/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.584021:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.584023:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.584054:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076996608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768524.584060:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996736:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768524.584088:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.584098:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350932300 x1862023076996736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.584107:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023076996736/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.584115:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.584117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.584120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996736:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768524.584125:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768524.584142:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.584150:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350931880 x1862023076996864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.584161:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023076996864/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.584168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.584171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.584174:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768524.584180:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996992:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768524.584196:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.584205:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350932a00 x1862023076996992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.584215:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023076996992/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.584222:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.584224:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.584227:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076996992:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768524.584232:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076997120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768524.584248:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768524.584257:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768524/real 1775768524] req@ffffa04350930700 x1862023076997120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768524.584267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023076997120/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768524.584274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768524.584277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768524.584279:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076997120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768524.775135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768524.775135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768525.607176:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768525.607292:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768525.799136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768525.799136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768526.631135:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768526.631191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768526.823126:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768526.823126:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768527.655113:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768527.655153:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768527.847127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768527.847127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768528.679121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768528.679171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768528.871093:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768528.871095:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768529.703140:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768529.703142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768529.703149:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768529.703154:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703163:0:52650:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703175:0:52650:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775768529.703179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768529.703182:0:52650:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768529.703221:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023076997248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768529.703248:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703251:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768529.703253:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703257:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703261:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703264:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703273:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768529.703285:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023076997376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768529.703292:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703294:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768529.703295:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703298:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703300:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703303:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703306:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768529.703313:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023076997504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768529.703320:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076997248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768529.703322:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703328:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768529.703330:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703334:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703337:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703340:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703345:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768529.703357:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023076997632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768529.703366:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703369:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768529.703370:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703375:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703379:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703382:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703387:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1775768529.703395:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768529.703396:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023076997760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768529.703403:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703405:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768529.703407:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703410:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703413:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703415:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703418:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768529.703428:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023076997888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768529.703436:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703440:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775768529.703440:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349641500 x1862023076997248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768529.703442:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703449:0:52650:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703455:0:52650:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703460:0:52650:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703465:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768529.703469:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023076997248/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768529.703478:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023076998016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768529.703482:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703491:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768529.703494:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768529.703498:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076997248:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768529.703499:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768529.703503:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768529.703509:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997376:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768529.703510:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703517:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703521:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768529.703526:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768529.703541:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023076998144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768529.703541:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768529.703552:0:52650:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768529.703556:0:52650:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768529.703557:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349640380 x1862023076997376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768529.703558:0:52650:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768529.703567:0:52650:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5031 00000100:00080000:0.0:1775768529.703574:0:52650:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768529.703580:0:52650:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023076997376/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768529.703587:0:52650:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768529.703589:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703594:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997376:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768529.703613:0:52650:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023076998272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768529.703630:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768529.703647:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703656:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349642680 x1862023076997504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703664:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023076997504/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703669:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703673:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768529.703677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997632:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768529.703693:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703702:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349640e00 x1862023076997632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703712:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023076997632/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703732:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703735:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997632:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768529.703741:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768529.703757:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703766:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349642300 x1862023076997760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703775:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023076997760/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703780:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703784:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076997760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768529.703787:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076997888:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768529.703798:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703811:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349642a00 x1862023076997888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703818:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023076997888/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703824:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076997888:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768529.703829:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768529.703842:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703850:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349643100 x1862023076998016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703860:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023076998016/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703882:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768529.703894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998144:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768529.703907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703914:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349640a80 x1862023076998144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703920:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023076998144/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703925:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703926:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998144:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768529.703931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768529.703939:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768529.703943:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768529/real 1775768529] req@ffffa04349641180 x1862023076998272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768529.703949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023076998272/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768529.703953:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768529.703955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768529.703957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076998272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768529.895134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768529.895134:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768530.727173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768530.727245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768530.919145:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768530.919145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768531.751137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768531.751159:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768531.943156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768531.943156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768532.775125:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768532.775229:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768532.967135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768532.967176:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768533.799128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768533.799161:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768533.991157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768533.991389:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768534.823132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768534.823151:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823157:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768534.823160:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823170:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823178:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823186:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768534.823218:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076998400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823236:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823240:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768534.823242:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823246:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823250:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823254:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:1.0:1775768534.823256:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768534.823259:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076998400:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768534.823264:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768534.823277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023076998528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823286:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823290:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768534.823291:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823294:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823310:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823314:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823319:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768534.823329:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023076998656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823337:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823341:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768534.823343:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823347:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823350:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823353:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823358:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768534.823366:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023076998784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823373:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823376:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768534.823378:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823381:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823384:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823389:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768534.823390:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823393:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768534.823402:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023076998912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823411:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823414:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768534.823416:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823420:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823423:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823426:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823430:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768534.823438:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023076999040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823450:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768534.823451:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823454:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823457:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823460:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823464:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768534.823474:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7e680 x1862023076998400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768534.823475:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023076999168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823506:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823509:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768534.823511:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823516:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00100000:3.0:1775768534.823517:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076998400/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768534.823521:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768534.823528:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768534.823533:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768534.823534:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768534.823539:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823543:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076998400:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768534.823564:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023076999296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768534.823576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768534.823579:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768534.823581:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768534.823586:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5036 00000100:00080000:2.0:1775768534.823590:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775768534.823590:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998528:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768534.823595:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768534.823601:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768534.823614:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023076999424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768534.823627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7e300 x1862023076998528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823656:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023076998528/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823664:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823668:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823670:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998528:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768534.823676:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998656:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768534.823697:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823707:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7c380 x1862023076998656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823718:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023076998656/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823725:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823728:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823731:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768534.823736:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998784:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768534.823751:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823759:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7f480 x1862023076998784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823770:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023076998784/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823779:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823782:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823785:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998784:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768534.823790:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768534.823806:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823815:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7f100 x1862023076998912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823826:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023076998912/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823834:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823837:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823840:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076998912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768534.823845:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999040:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768534.823872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7dc00 x1862023076999040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023076999040/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999040:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768534.823910:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768534.823924:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.823933:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7d500 x1862023076999168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.823949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023076999168/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.823955:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.823958:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.823961:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768534.823978:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999296:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768534.823996:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.824005:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7c000 x1862023076999296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.824017:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023076999296/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.824025:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.824045:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.824049:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999296:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768534.824068:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768534.824086:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768534.824095:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768534/real 1775768534] req@ffffa0434aa7e680 x1862023076999424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768534.824105:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023076999424/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768534.824118:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768534.824121:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768534.824124:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023076999424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768535.015136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768535.015137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768535.847186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768535.847236:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768536.039101:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768536.039106:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768536.871181:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768536.871311:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768537.063184:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768537.063189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768537.895216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768537.895307:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768538.087183:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768538.087832:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:1.0:1775768538.919131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768538.919209:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768539.111180:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768539.111180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768539.943116:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768539.943127:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943134:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768539.943137:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943159:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943169:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943176:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768539.943205:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023076999552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943238:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943242:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768539.943244:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943248:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943252:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943256:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943264:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768539.943272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076999552:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768539.943277:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023076999680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943286:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943289:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768539.943291:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943293:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943296:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943298:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943302:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768539.943311:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023076999808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943317:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943320:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768539.943321:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943324:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943327:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943329:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943333:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768539.943334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768539.943342:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023076999936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943348:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943351:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768539.943352:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943354:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943356:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943359:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943361:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768539.943371:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7fb80 x1862023076999552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768539.943377:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023077000064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943400:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768539.943402:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943405:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00100000:1.0:1775768539.943409:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023076999552/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768539.943409:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943417:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00010000:00010000:0.0:1775768539.943419:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768539.943426:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768539.943426:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768539.943435:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943440:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023076999552:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768539.943443:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023077000192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943455:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943458:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768539.943460:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943465:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00100000:1.0:1775768539.943467:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768539.943470:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943476:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943481:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768539.943492:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023077000320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768539.943512:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768539.943522:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768539.943525:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768539.943526:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768539.943531:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00100000:1.0:1775768539.943534:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7ce00 x1862023076999680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768539.943536:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768539.943543:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943551:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768539.943559:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023076999680/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768539.943562:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023077000448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768539.943584:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768539.943585:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768539.943588:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768539.943590:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768539.943594:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768539.943596:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5041 00000100:00080000:2.0:1775768539.943601:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768539.943603:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999680:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768539.943606:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768539.943610:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768539.943617:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023077000576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768539.943626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768539.943649:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7df80 x1862023076999808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943673:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023076999808/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943680:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943683:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943686:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768539.943691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999936:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768539.943710:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943719:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7ed80 x1862023076999936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943730:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023076999936/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943738:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943741:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943744:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023076999936:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768539.943750:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768539.943766:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943776:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7d880 x1862023077000064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943788:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023077000064/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943798:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943801:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768539.943806:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000192:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768539.943823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7ea00 x1862023077000192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023077000192/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943850:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943853:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943856:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000192:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768539.943873:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768539.943890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943899:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7f800 x1862023077000320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943910:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023077000320/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768539.943933:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000448:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768539.943950:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.943959:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7c700 x1862023077000448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.943969:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023077000448/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.943976:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.943979:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.943982:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000448:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768539.943987:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768539.944005:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768539.944014:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768539/real 1775768539] req@ffffa0434aa7fb80 x1862023077000576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768539.944024:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023077000576/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768539.944049:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768539.944052:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768539.944056:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077000576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768540.135144:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768540.135144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768540.967134:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768540.967191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768541.159364:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768541.160333:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768541.991121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768541.991209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768542.183149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768542.183661:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768543.015259:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:0.0:1775768543.015393:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768543.207153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768543.207153:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768544.042491:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:2.0:1775768544.042578:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768544.231125:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768544.231125:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768545.063125:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768545.063128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768545.063133:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768545.063137:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063146:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063155:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1775768545.063169:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768545.063211:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768545.063250:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077000704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063270:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063274:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768545.063275:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063279:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063284:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768545.063289:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063297:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768545.063299:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077000704:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768545.063312:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077000832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063320:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063322:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768545.063323:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063325:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063328:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768545.063331:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063335:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768545.063342:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077000960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063347:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063350:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768545.063351:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063352:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063355:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768545.063358:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768545.063361:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063365:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768545.063372:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023077001088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063378:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063380:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768545.063381:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063384:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063386:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768545.063389:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063392:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768545.063394:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349643b80 x1862023077000704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768545.063402:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077001216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063408:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063411:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775768545.063412:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077000704/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768545.063414:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063422:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063425:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768545.063425:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768545.063429:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063432:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1775768545.063432:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063437:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077000704:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768545.063454:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077001344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768545.063461:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063463:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768545.063464:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063467:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063470:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768545.063472:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768545.063475:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063480:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768545.063490:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077001472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768545.063496:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768545.063499:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768545.063501:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768545.063503:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063506:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00080000:0.0:1775768545.063508:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768545.063509:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349643800 x1862023077000832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768545.063514:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063520:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768545.063525:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077000832/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768545.063530:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077001600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768545.063535:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768545.063541:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768545.063541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768545.063546:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775768545.063548:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000832:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768545.063549:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768545.063555:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5046 00000100:00100000:1.0:1775768545.063557:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000960:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768545.063558:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768545.063563:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768545.063567:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768545.063574:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077001728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768545.063577:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063587:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349641880 x1862023077000960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063595:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077000960/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063607:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063611:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063613:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077000960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768545.063617:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001088:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768545.063628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063633:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349641c00 x1862023077001088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063640:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023077001088/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063644:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001088:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768545.063650:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768545.063658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063663:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349640700 x1862023077001216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063679:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077001216/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063686:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063688:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768545.063691:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001344:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768545.063700:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349641180 x1862023077001344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077001344/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063718:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063721:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063724:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001344:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768545.063742:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768545.063759:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063769:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349640a80 x1862023077001472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063779:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077001472/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063791:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063794:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063797:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768545.063802:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001600:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768545.063823:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063832:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349643100 x1862023077001600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063841:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077001600/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063856:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063859:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001600:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768545.063866:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768545.063883:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768545.063892:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768545/real 1775768545] req@ffffa04349642a00 x1862023077001728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768545.063902:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077001728/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768545.063908:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768545.063910:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768545.063920:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077001728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768545.255094:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768545.255132:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768546.087137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768546.087175:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768546.279136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768546.279136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768547.111207:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768547.111269:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768547.303150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768547.303150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768548.135173:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768548.135268:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768548.327187:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768548.327188:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768549.159235:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768549.159306:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768549.351238:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768549.351300:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768550.183136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768550.183161:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183166:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768550.183169:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183179:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183188:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183195:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768550.183233:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023077001856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183266:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183270:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768550.183271:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183275:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183279:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183283:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183291:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768550.183305:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023077001984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183320:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183323:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768550.183325:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183328:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183331:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183335:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183339:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768550.183346:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077001856:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768550.183348:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023077002112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183356:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183359:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768550.183360:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183363:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183366:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183369:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183373:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768550.183380:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023077002240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183387:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768550.183402:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183407:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183411:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183414:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183524:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768550.183542:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023077002368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183561:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768550.183562:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768550.183565:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768550.183566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183575:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183579:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183588:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183593:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768550.183602:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023077002496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183611:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183614:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768550.183615:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768550.183618:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7e680 x1862023077001856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768550.183618:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183621:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183625:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183628:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768550.183636:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023077002624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183657:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768550.183658:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023077001856/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768550.183660:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768550.183661:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768550.183665:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:0.0:1775768550.183667:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768550.183667:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768550.183671:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768550.183671:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768550.183674:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077001856:192.168.202.158@tcp:250:kworker.0 00000008:00100000:2.0:1775768550.183675:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768550.183681:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768550.183684:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023077002752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768550.183691:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768550.183694:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768550.183695:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768550.183699:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768550.183699:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5051 00000100:00080000:2.0:1775768550.183702:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768550.183705:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768550.183709:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768550.183710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7c000 x1862023077001984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768550.183717:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023077002880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768550.183720:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023077001984/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.183728:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.183731:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.183734:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077001984:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768550.183771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768550.183802:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.183813:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7d500 x1862023077002112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.183823:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023077002112/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.183831:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.183834:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.183837:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768550.183842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002240:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768550.183857:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.183866:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7dc00 x1862023077002240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.183876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023077002240/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.183885:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.183888:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.183891:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002240:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768550.183897:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768550.183913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.183922:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7f100 x1862023077002368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.183932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023077002368/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.183951:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.183955:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.183958:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077002368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768550.183963:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002496:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768550.183979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.183988:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7f480 x1862023077002496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.183998:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023077002496/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.184006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.184009:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.184012:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002496:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768550.184017:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002624:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768550.184052:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.184062:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7c380 x1862023077002624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.184072:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023077002624/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.184080:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.184083:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.184086:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768550.184092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002752:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768550.184109:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.184118:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7e300 x1862023077002752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.184128:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023077002752/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.184135:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.184138:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.184142:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002752:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768550.184147:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768550.184161:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768550.184168:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768550/real 1775768550] req@ffffa0434aa7fb80 x1862023077002880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768550.184178:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023077002880/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768550.184185:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768550.184188:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768550.184190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077002880:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768550.184735:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768550.377017:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768550.377649:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768551.207444:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768551.207559:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768551.399171:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768551.399501:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768552.231115:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768552.231179:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768552.423148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768552.423153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768553.255159:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768553.255273:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768553.447134:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768553.449718:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768554.279085:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768554.279112:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768554.471175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768554.471813:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768555.303140:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768555.303159:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303166:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768555.303170:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303180:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5056 00010000:00010000:3.0:1775768555.303187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768555.303190:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303225:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768555.303261:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023077003008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768555.303289:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303293:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768555.303294:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303306:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:0.0:1775768555.303313:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303318:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768555.303327:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768555.303348:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077003008:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768555.303360:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023077003136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768555.303372:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303375:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768555.303377:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303381:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:0.0:1775768555.303384:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303387:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768555.303391:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768555.303400:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023077003264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768555.303406:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303408:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768555.303410:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303415:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5056 00000800:00000100:1.0:1775768555.303415:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768555.303420:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303422:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768555.303434:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768555.303445:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023077003392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768555.303453:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303456:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768555.303457:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303460:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:0.0:1775768555.303464:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303469:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768555.303471:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349642300 x1862023077003008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768555.303475:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768555.303489:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023077003520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768555.303498:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023077003008/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768555.303500:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303507:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768555.303510:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303518:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:1.0:1775768555.303519:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768555.303521:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303528:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768555.303528:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775768555.303532:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768555.303537:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077003008:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768555.303545:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077003648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768555.303552:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303554:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768555.303556:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303559:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:0.0:1775768555.303561:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303562:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768555.303565:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768555.303570:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077003776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768555.303576:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003136:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768555.303577:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303596:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768555.303599:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303603:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5056 00000800:00000100:1.0:1775768555.303607:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768555.303608:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303614:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768555.303619:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768555.303624:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349640e00 x1862023077003136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768555.303633:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077003904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768555.303644:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023077003136/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768555.303645:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768555.303651:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768555.303653:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768555.303663:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5056 00000100:00080000:0.0:1775768555.303669:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768555.303675:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:1.0:1775768555.303682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768555.303683:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1775768555.303689:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.303693:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003136:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768555.303697:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077004032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768555.303704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768555.303731:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.303743:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349642680 x1862023077003264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.303754:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023077003264/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.303761:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.303763:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.303766:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003264:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768555.303771:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003392:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768555.303789:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.303799:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349640380 x1862023077003392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304087:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023077003392/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304096:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304099:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304112:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003392:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768555.304118:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768555.304143:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.304156:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349641500 x1862023077003520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304166:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023077003520/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304174:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304177:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304179:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077003520:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768555.304185:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003648:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768555.304216:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.304225:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349642a00 x1862023077003648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304236:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077003648/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304243:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304246:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304249:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003648:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768555.304268:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003776:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768555.304283:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.304291:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349643100 x1862023077003776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077003776/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304311:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304313:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768555.304318:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003904:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768555.304330:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.304338:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349640a80 x1862023077003904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304361:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077003904/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304367:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304370:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304372:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077003904:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768555.304377:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768555.304392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768555.304400:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768555/real 1775768555] req@ffffa04349641180 x1862023077004032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768555.304410:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077004032/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768555.304419:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768555.304421:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768555.304425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768555.495430:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768555.496091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768556.327138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768556.327339:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768556.519138:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768556.519190:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768557.351177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768557.351252:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768557.543170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768557.543675:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768558.375788:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768558.375815:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768558.567130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768558.567136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768559.399192:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768559.399356:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768559.591127:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768559.591140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768560.423192:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768560.423241:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768560.423308:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423313:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768560.423317:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423325:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423334:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423339:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768560.423369:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023077004160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768560.423397:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423400:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768560.423402:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423406:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423409:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423413:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423422:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768560.423436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077004160:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768560.423436:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023077004288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768560.423452:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423456:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768560.423457:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423461:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423465:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423468:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423473:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768560.423482:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023077004416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768560.423492:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768560.423493:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423497:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768560.423499:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423504:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423507:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423511:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423516:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768560.423524:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7c700 x1862023077004160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768560.423528:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023077004544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768560.423548:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423551:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768560.423553:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423558:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423562:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423564:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423568:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768560.423572:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023077004160/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768560.423579:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023077004672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768560.423584:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768560.423591:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768560.423593:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768560.423598:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1775768560.423600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077004160:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768560.423601:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423608:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423612:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423615:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423619:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768560.423622:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768560.423633:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023077004800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768560.423642:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768560.423644:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423648:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768560.423650:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768560.423653:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7f800 x1862023077004288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768560.423657:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423663:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423669:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423676:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768560.423677:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023077004288/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423685:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423690:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768560.423691:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023077004928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768560.423693:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004288:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768560.423702:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768560.423709:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768560.423713:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768560.423716:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768560.423720:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423724:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768560.423727:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423732:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1775768560.423733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423745:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7ea00 x1862023077004416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768560.423755:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023077005056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768560.423759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023077004416/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768560.423775:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768560.423778:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768560.423779:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768560.423782:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768560.423786:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00080000:2.0:1775768560.423789:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5062 00000100:00080000:2.0:1775768560.423792:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768560.423799:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:2.0:1775768560.423799:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768560.423806:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768560.423811:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768560.423817:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023077005184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768560.423831:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423840:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7d880 x1862023077004544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.423847:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023077004544/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423856:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.423860:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004544:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768560.423863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004672:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768560.423872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423877:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7ed80 x1862023077004672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.423891:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023077004672/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423896:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423898:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.423900:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077004672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768560.423911:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004800:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768560.423921:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423926:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7df80 x1862023077004800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.423932:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023077004800/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423937:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423938:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.423940:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004800:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768560.423943:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004928:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768560.423951:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7ce00 x1862023077004928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.423961:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023077004928/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423965:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423967:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.423968:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077004928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768560.423971:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005056:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768560.423979:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.423983:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7c700 x1862023077005056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.423989:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023077005056/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.423993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.423995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.423996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005056:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768560.424000:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005184:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768560.424012:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768560.424021:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768560/real 1775768560] req@ffffa0434aa7fb80 x1862023077005184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768560.424057:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023077005184/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768560.424064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768560.424067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768560.424070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768560.615164:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768560.615350:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:1.0:1775768561.447137:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768561.447255:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768561.639136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768561.639138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768562.471320:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768562.471380:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:3.0:1775768562.663401:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768562.663536:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768563.496115:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768563.496221:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768563.687130:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768563.687216:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768564.519203:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768564.519318:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768564.711137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768564.711137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768565.543117:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768565.543121:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543128:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1775768565.543130:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768565.543132:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543141:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543150:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543156:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768565.543190:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077005312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543211:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543213:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768565.543215:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543217:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543220:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543223:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543230:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768565.543250:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077005440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543261:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768565.543263:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077005312:192.168.202.158@tcp:250:kworker.0 00000100:00080000:1.0:1775768565.543264:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768565.543266:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543273:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543275:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543278:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543282:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768565.543291:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077005568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543297:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543299:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768565.543300:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543302:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543317:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543319:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543322:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768565.543328:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077005696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543333:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543335:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768565.543337:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543339:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543341:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543343:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543346:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768565.543354:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077005824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543359:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543361:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768565.543362:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543365:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543367:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768565.543369:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005440:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768565.543372:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543379:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768565.543389:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077005952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543396:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543398:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768565.543399:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543402:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543404:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768565.543407:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005568:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768565.543409:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543415:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768565.543425:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077006080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543431:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543433:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768565.543434:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768565.543437:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005696:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768565.543439:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543442:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768565.543444:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543446:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768565.543458:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077006208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768565.543464:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768565.543466:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768565.543468:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768565.543471:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5067 00000100:00080000:1.0:1775768565.543474:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768565.543475:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005824:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768565.543479:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768565.543484:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768565.543496:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077006336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768565.543504:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768565.543524:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768565.543538:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768565.543555:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768565.735122:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768565.735123:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768566.567136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768566.567252:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768566.759155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768566.759155:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768567.591106:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768567.591193:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768567.783162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768567.783451:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768568.615131:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775768568.615438:0:2395:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775768568.615475:0:2395:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768568.615490:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775768568.615585:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350930700 x1862023077005312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768590 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768568.615628:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077005312/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768590 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768568.615638:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768568.615643:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768568.615647:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077005312:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768568.615879:0:2395:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768568.615949:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350930000 x1862023077006336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768568.615965:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077006336/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768568.615973:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768568.615977:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768568.615980:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768568.616091:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616109:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616120:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616127:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616135:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616141:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768568.616148:0:2395:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775768568.616166:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350932680 x1862023077005952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616179:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077005952/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616191:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616194:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077005952:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768568.616201:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350931c00 x1862023077006080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616222:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077006080/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616230:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616233:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616236:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768568.616241:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350930a80 x1862023077006208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616251:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077006208/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616259:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616261:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616265:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077006208:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768568.616271:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350932a00 x1862023077005440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616281:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077005440/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616287:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616291:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616294:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005440:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768568.616299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350931880 x1862023077005568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616310:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077005568/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616317:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616320:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616324:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768568.616330:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350932300 x1862023077005696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616340:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077005696/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616347:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616349:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005696:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768568.616359:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768565/real 1775768568] req@ffffa04350930e00 x1862023077005824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768568.616375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077005824/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768568.616383:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768568.616386:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768568.616389:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077005824:192.168.202.158@tcp:8:kworker.0 00010000:00010000:2.0:1775768568.618015:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768568.807105:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768568.807109:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768569.639136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768569.639213:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768569.831148:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768569.831342:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768570.663206:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.663216:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768570.663219:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.663228:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.663237:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.663243:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768570.663272:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077006464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.663343:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077006464:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768570.663402:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768570.663430:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.663434:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768570.663435:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.663440:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.663444:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.663447:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.663456:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768570.663469:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077006592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.664260:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825dc00 x1862023077006464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.664282:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077006464/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.664292:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.664297:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.664300:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077006464:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768570.664334:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006592:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768570.664370:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768570.664386:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825d500 x1862023077006592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.664405:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077006592/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.664414:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.664417:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.664420:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006592:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768570.665360:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.665366:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768570.665369:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.665375:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.665383:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.665389:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.665398:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768570.665422:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077006720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.665479:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768570.665515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768570.665648:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768570.665904:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.665908:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768570.665911:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.665915:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.665920:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.665924:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.665930:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768570.665943:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077006848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.666297:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825e300 x1862023077006720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.666314:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077006720/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.666323:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.666326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.666330:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006720:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768570.666349:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006848:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768570.666376:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768570.666388:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825e680 x1862023077006848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.666399:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077006848/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.666411:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.666415:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.666418:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006848:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768570.666538:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.666542:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768570.666544:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.666548:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.666552:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.666555:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.666561:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768570.666575:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077006976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.666623:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768570.666800:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768570.666823:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.666826:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768570.666828:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.666833:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.666837:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.666841:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.666847:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768570.666860:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077007104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.666900:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825ea00 x1862023077006976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.666913:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077006976/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.666921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.666925:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.666928:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077006976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768570.666940:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007104:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768570.666962:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768570.666974:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825f100 x1862023077007104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.666985:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077007104/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.666993:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.666996:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.667000:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007104:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768570.667155:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.667158:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768570.667160:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.667164:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.667169:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.667172:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.667177:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768570.667202:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077007232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.667247:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768570.667269:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768570.667290:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.667294:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768570.667296:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.667300:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.667304:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.667308:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.667313:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768570.667324:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077007360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.667363:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825c000 x1862023077007232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.667376:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077007232/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.667384:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.667386:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.667390:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768570.667402:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007360:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768570.667422:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768570.667434:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825ed80 x1862023077007360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.667444:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077007360/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.667452:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.667455:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.667458:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007360:192.168.202.158@tcp:8:kworker.0 00000100:00080000:3.0:1775768570.667550:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768570.667554:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768570.667556:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768570.667561:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5072 00000100:00080000:3.0:1775768570.667565:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768570.667569:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768570.667575:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768570.667587:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077007488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768570.667634:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768570.667719:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00010000:00010000:2.0:1775768570.667853:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768570.667891:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768570/real 1775768570] req@ffffa0435825ce00 x1862023077007488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768570.667905:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077007488/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768570.667912:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768570.667915:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768570.667918:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077007488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768570.855148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768570.855278:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768571.687195:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768571.687604:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768571.879147:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768571.879351:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768572.711156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768572.711193:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768572.903155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768572.903558:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768573.735128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768573.735231:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768573.927119:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768573.927119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768574.759257:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768574.806652:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768574.951165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768574.952302:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768575.783220:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768575.783221:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768575.783231:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768575.783236:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.783247:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783259:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.783266:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768575.783300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077007616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.783327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783331:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768575.783332:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.783337:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783341:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.783345:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.783354:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768575.783370:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023077007744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768575.783379:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077007616:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768575.783384:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783388:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768575.783390:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.783395:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783400:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.783404:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.783409:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768575.783423:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077007872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.783432:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783436:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775768575.783440:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768575.783441:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.783447:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783452:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.783457:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.783462:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768575.783478:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349640700 x1862023077007616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768575.783479:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077008000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.783513:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783517:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768575.783519:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768575.783521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077007616/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768575.783530:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783538:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768575.783540:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768575.783544:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768575.783550:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000008:00100000:0.0:1775768575.783551:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768575.783556:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077007616:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768575.783566:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077008128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.783575:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783579:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768575.783581:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768575.783584:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077007744:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768575.783588:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.783595:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.783599:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.783605:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768575.783618:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077008256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768575.783627:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768575.783632:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.783636:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768575.783638:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.783642:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00100000:3.0:1775768575.784454:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349641c00 x1862023077007744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.784466:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023077007744/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.784472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.784474:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.784477:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077007744:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768575.784482:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077007872:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768575.788222:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.788241:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.788256:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768575.788294:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077008384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.788314:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.788317:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768575.788320:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.788325:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.788330:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.788334:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.788338:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768575.788349:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077008512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768575.788357:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768575.788360:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768575.788362:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768575.788365:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5077 00000100:00080000:0.0:1775768575.788369:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768575.788372:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768575.788377:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768575.788387:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077008640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768575.791229:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791273:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349641880 x1862023077007872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791299:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077007872/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791310:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791315:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791319:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077007872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768575.791329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008000:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768575.791373:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791385:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349643800 x1862023077008000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791401:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077008000/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791409:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791412:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791416:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008000:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768575.791422:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768575.791443:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349643b80 x1862023077008128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791464:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077008128/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791478:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768575.791510:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008256:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768575.791531:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791542:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349640700 x1862023077008256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791554:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077008256/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791584:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008256:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768575.791593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768575.791613:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349641180 x1862023077008384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077008384/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791642:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791648:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768575.791654:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008512:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768575.791672:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791682:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349640a80 x1862023077008512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791692:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077008512/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791700:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791703:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791706:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008512:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768575.791713:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768575.791731:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768575.791741:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768575/real 1775768575] req@ffffa04349643100 x1862023077008640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768575.791752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077008640/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768575.791759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768575.791762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768575.791765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077008640:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768575.848111:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768575.975205:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768575.975365:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768576.807196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768576.871235:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768576.999157:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768576.999161:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768577.831153:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768577.895377:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768578.023168:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768578.023190:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768578.855162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768578.919277:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768579.047124:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768579.047158:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768579.879256:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768579.943478:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768580.071191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768580.071251:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768580.903171:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768580.903313:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.903319:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768580.903323:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903332:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903344:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.903350:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768580.903672:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077008768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.903706:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.903711:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768580.903713:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903717:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903723:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.903729:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.903741:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768580.903748:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077008768:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768580.903756:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023077008896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.903767:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.903771:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768580.903773:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903777:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903781:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.903786:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.903791:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768580.903804:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023077009024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.903814:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.903818:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768580.903820:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903824:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903828:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768580.903829:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768580.903835:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.903841:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768580.903854:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023077009152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.903864:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.903868:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768580.903870:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903874:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903878:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.903881:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.903886:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768580.903887:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349642a00 x1862023077008768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768580.903901:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023077009280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768580.903908:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077008768/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.903918:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.903922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:0.0:1775768580.903925:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768580.903926:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077008768:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768580.903932:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768580.903935:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903940:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.903944:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.903948:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768580.903952:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008896:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768580.903954:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768580.903970:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023077009408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.903981:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768580.903983:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768580.903987:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768580.903990:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.903995:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00100000:3.0:1775768580.903997:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349641500 x1862023077008896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768580.904001:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.904009:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.904016:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768580.904018:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023077008896/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904048:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768580.904053:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077009536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768580.904054:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904060:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077008896:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768580.904072:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009024:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768580.904073:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775768580.904107:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904118:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349640380 x1862023077009024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904130:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023077009024/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904143:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904146:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904149:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768580.904155:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009152:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768580.904176:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904187:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349642680 x1862023077009152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904195:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023077009152/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904206:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904209:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904211:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009152:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768580.904215:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768580.904228:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904234:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349640e00 x1862023077009280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904241:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023077009280/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904246:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904248:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904250:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077009280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768580.904264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009408:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768580.904273:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904279:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349642300 x1862023077009408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904286:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023077009408/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904290:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904292:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904294:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009408:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768580.904297:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768580.904305:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904310:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349642a00 x1862023077009536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904319:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077009536/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904327:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904330:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904332:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009536:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768580.904791:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768580.904792:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.904796:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.904800:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.904802:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.904807:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768580.904816:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077009664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768580.904831:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768580.904833:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768580.904835:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768580.904837:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5082 00000100:00080000:0.0:1775768580.904839:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768580.904841:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768580.904844:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768580.904850:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077009792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768580.904875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009664:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768580.904895:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904906:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349643100 x1862023077009664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904915:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077009664/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904921:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904923:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009664:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768580.904929:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768580.904952:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768580.904962:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768580/real 1775768580] req@ffffa04349640a80 x1862023077009792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768580.904974:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077009792/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768580.904982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768580.904985:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768580.904988:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077009792:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768580.967269:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768581.095193:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768581.095194:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768581.927168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768581.991187:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768582.119154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768582.119155:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768582.951122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768583.015186:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768583.143151:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768583.143151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768583.975107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768584.039178:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768584.167119:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768584.167123:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768584.999151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768585.063163:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768585.191136:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768585.191136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768586.023142:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023158:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768586.023162:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768586.023163:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768586.023173:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023183:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023189:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768586.023225:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023077009920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023255:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023259:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768586.023261:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023265:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023269:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023274:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023283:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768586.023292:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077009920:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768586.023300:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077010048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023311:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023315:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768586.023317:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023320:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023324:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023329:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023334:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768586.023344:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077010176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023353:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768586.023356:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768586.023358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768586.023360:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023364:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023368:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023373:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023378:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768586.023388:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077010304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023397:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768586.023399:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825d880 x1862023077009920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768586.023412:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768586.023415:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768586.023420:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023077009920/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768586.023423:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023429:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023436:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768586.023439:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775768586.023441:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1775768586.023446:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023451:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077009920:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768586.023455:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077010432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023464:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023468:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768586.023470:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023474:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00100000:0.0:1775768586.023476:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010048:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768586.023479:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023486:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023490:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768586.023502:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077010560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768586.023508:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.023523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825ce00 x1862023077010048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768586.023530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023537:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768586.023539:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768586.023543:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077010048/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768586.023546:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023552:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023558:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023567:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768586.023569:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023575:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023581:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010048:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768586.023581:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077010688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768586.023593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010176:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768586.023597:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768586.023602:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023607:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023611:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768586.023616:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768586.023616:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023623:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768586.023633:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077010816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768586.023649:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768586.023652:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768586.023654:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768586.023657:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5087 00000100:00080000:3.0:1775768586.023660:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768586.023664:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768586.023667:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768586.023675:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077010944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768586.023746:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825ed80 x1862023077010176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.023759:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077010176/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.023766:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023769:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023772:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768586.023778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010304:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768586.023797:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.023806:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825c000 x1862023077010304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.023816:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077010304/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.023822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023825:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023828:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010304:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768586.023833:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768586.023847:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.023855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825f100 x1862023077010432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.023864:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077010432/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.023872:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023874:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023877:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077010432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768586.023892:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010560:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768586.023907:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.023916:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825ea00 x1862023077010560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.023926:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077010560/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.023933:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023936:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023939:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010560:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768586.023944:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768586.023957:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.023965:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825e680 x1862023077010688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.023975:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077010688/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.023982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.023984:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.023987:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768586.023991:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010816:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768586.024004:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.024012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825e300 x1862023077010816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.024021:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077010816/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.024057:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.024060:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.024063:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010816:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768586.024068:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768586.024085:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768586.024094:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768586/real 1775768586] req@ffffa0435825d500 x1862023077010944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768586.024104:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077010944/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768586.024111:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768586.024114:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768586.024117:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077010944:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768586.087264:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768586.215149:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768586.215149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768587.047130:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768587.111182:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768587.239140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768587.239140:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768588.071127:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768588.135165:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768588.263136:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768588.263141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768589.095122:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768589.159183:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768589.287144:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768589.287242:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768590.119527:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768590.183182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768590.311129:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768590.311129:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768591.143129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768591.143143:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143149:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768591.143153:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143162:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143170:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143176:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768591.143206:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077011072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768591.143231:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143234:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768591.143236:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143240:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143244:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143249:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768591.143258:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768591.143264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077011072:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768591.143271:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077011200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768591.143279:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143281:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768591.143282:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143285:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143288:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143290:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768591.143294:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768591.143301:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077011328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768591.143307:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143311:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768591.143312:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143315:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143320:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775768591.143320:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768591.143327:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768591.143332:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768591.143343:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077011456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768591.143353:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775768591.143357:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349641180 x1862023077011072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768591.143358:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768591.143361:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143369:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143377:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143383:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768591.143388:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768591.143389:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077011072/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143401:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775768591.143405:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077011584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768591.143407:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143414:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077011072:192.168.202.158@tcp:250:kworker.0 00000100:00080000:0.0:1775768591.143419:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143422:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768591.143424:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143428:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143432:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143436:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143440:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011200:192.168.202.158@tcp:38:kworker.0 00000008:00100000:0.0:1775768591.143451:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768591.143466:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023077011712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775768591.143471:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:0.0:1775768591.143481:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143483:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768591.143485:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775768591.143488:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349640700 x1862023077011200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768591.143492:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143499:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143504:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143508:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077011200/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768591.143510:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1775768591.143522:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143525:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768591.143529:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077011840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768591.143532:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011200:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768591.143545:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011328:192.168.202.158@tcp:38:kworker.0 00000100:00080000:0.0:1775768591.143549:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143552:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768591.143553:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143558:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143561:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143564:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000800:00000100:3.0:1775768591.143568:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000008:00100000:0.0:1775768591.143569:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768591.143579:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077011968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768591.143582:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349643b80 x1862023077011328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768591.143602:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768591.143605:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775768591.143607:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077011328/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768591.143608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768591.143617:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5092 00000100:00080000:0.0:1775768591.143624:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768591.143631:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:3.0:1775768591.143631:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775768591.143638:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1775768591.143638:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143643:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768591.143652:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077012096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768591.143653:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011456:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768591.143678:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143687:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349643800 x1862023077011456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143697:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077011456/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143704:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143707:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143709:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011456:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768591.143715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768591.143728:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143735:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349641880 x1862023077011584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143745:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077011584/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143754:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143757:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077011584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768591.143773:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011712:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768591.143788:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349641c00 x1862023077011712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143806:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023077011712/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143813:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143816:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143819:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011712:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768591.143825:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768591.143840:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143849:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349641180 x1862023077011840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143859:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077011840/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143865:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143868:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143870:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768591.143875:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011968:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768591.143888:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143896:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349640a80 x1862023077011968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143906:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077011968/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143914:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143917:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143919:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077011968:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768591.143924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768591.143937:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768591.143945:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768591/real 1775768591] req@ffffa04349643100 x1862023077012096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768591.143955:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077012096/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768591.143962:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768591.143965:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768591.143967:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012096:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768591.207178:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768591.335139:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768591.335139:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768592.167120:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768592.231166:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768592.359245:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768592.359245:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768593.191166:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768593.255170:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768593.383236:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768593.383250:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768594.215128:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768594.279153:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768594.407131:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768594.407131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768595.239182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768595.303319:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768595.431140:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768595.431148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768596.263196:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768596.263348:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263355:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768596.263360:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263371:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263383:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263391:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768596.263437:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077012224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263471:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263475:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768596.263477:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263481:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263485:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263489:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263500:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768596.263512:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023077012352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263522:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263525:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768596.263527:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263531:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263534:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263538:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263542:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768596.263551:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023077012480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263560:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263563:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768596.263565:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263568:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263570:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263574:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263579:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768596.263590:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023077012608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263599:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263602:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768596.263604:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263607:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263610:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263614:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263619:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768596.263628:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023077012736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263636:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263639:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768596.263641:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263644:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263647:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263650:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263655:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768596.263664:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023077012864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263672:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263675:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768596.263677:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263681:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263685:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263688:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263692:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768596.263701:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077012992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263710:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263713:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768596.263715:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263718:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263720:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263723:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263728:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768596.263737:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077013120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768596.263746:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768596.263749:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768596.263751:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768596.263755:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5097 00000100:00080000:0.0:1775768596.263758:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768596.263761:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768596.263765:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768596.263774:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077013248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768596.263842:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077012224:192.168.202.158@tcp:250:kworker.0 00000800:00000100:3.0:1775768596.263913:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.263971:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349642a00 x1862023077012224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.263988:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077012224/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.263998:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264002:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264006:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077012224:192.168.202.158@tcp:250:kworker.0 00000100:00100000:3.0:1775768596.264012:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012352:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768596.264053:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264063:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349642300 x1862023077012352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264075:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023077012352/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264083:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264086:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264089:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012352:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768596.264095:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768596.264114:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264124:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349640e00 x1862023077012480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264135:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023077012480/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264142:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264145:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264148:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768596.264153:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012608:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768596.264170:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264180:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349642680 x1862023077012608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264192:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023077012608/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264200:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264203:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264205:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012608:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768596.264210:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768596.264227:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264236:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349640380 x1862023077012736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264254:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023077012736/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264263:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264266:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264269:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077012736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768596.264274:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012864:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768596.264291:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264299:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349641500 x1862023077012864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264310:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023077012864/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264317:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264320:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264323:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012864:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768596.264328:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:3.0:1775768596.264347:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264356:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349643100 x1862023077012992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264367:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077012992/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264380:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264383:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264386:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077012992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768596.264391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077013120:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768596.264409:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264419:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349640a80 x1862023077013120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264430:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077013120/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264437:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264440:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264443:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077013120:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768596.264448:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077013248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:3.0:1775768596.264465:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:3.0:1775768596.264475:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768596/real 1775768596] req@ffffa04349641180 x1862023077013248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768596.264486:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077013248/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768596.264494:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768596.264497:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:3.0:1775768596.264501:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077013248:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768596.327181:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768596.455191:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768596.455195:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768597.287597:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768597.354012:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768597.479157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768597.480466:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768598.311129:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768598.375171:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768598.503214:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768598.505283:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768599.335182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768599.399474:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768599.527175:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768599.527180:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768600.359132:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768600.423354:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768600.551135:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768600.551135:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768601.383138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768601.383403:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383409:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768601.383412:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383421:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383429:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383443:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768601.383475:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023077013376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383506:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383511:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768601.383512:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383516:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383520:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383523:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383538:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768601.383549:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077013504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383558:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383561:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768601.383563:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383566:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383569:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383573:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383577:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768601.383586:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077013632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383594:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383597:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768601.383598:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383601:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383603:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383606:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383609:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768601.383616:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077013760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383623:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383626:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768601.383628:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383631:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383633:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383636:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383639:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768601.383647:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077013888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383654:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383657:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768601.383658:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383661:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383663:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383666:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383670:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768601.383676:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077014016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383684:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383687:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768601.383688:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383691:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383694:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383696:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383700:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768601.383706:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077014144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383713:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383716:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768601.383718:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383720:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383723:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383726:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383730:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768601.383737:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077014272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768601.383745:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768601.383748:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768601.383749:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768601.383752:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5103 00000100:00080000:0.0:1775768601.383754:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768601.383757:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768601.383760:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768601.383767:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023077014400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768601.383829:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077013376:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775768601.383896:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.383931:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349641c00 x1862023077013376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.383947:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023077013376/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.383957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.383960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.383963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077013376:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768601.383971:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013504:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768601.383989:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.383999:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349641880 x1862023077013504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384009:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077013504/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384016:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384019:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384022:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013504:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768601.384059:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768601.384077:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384087:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349643800 x1862023077013632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384097:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077013632/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384104:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384107:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384110:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768601.384115:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013760:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768601.384143:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384151:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349643b80 x1862023077013760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384162:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077013760/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384168:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384171:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384174:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013760:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768601.384178:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768601.384190:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384198:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349640700 x1862023077013888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384208:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077013888/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384217:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384220:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384222:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077013888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768601.384227:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014016:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768601.384239:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384255:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349641180 x1862023077014016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384266:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077014016/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384280:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014016:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768601.384285:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768601.384299:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384307:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349640a80 x1862023077014144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384315:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077014144/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384321:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384324:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384327:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768601.384331:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014272:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768601.384345:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349643100 x1862023077014272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384362:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077014272/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384369:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384371:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384374:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014272:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768601.384378:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768601.384391:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768601.384398:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768601/real 1775768601] req@ffffa04349641500 x1862023077014400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768601.384407:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023077014400/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768601.384413:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768601.384415:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768601.384418:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077014400:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768601.447955:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768601.575141:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768601.575141:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768602.407121:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768602.471170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768602.599146:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768602.600975:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768603.431162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768603.495578:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768603.623436:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768603.623436:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768604.457967:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768604.520001:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768604.647163:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768604.647918:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768605.479916:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768605.543209:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768605.671137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768605.671138:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768606.503110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768606.504918:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.504925:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768606.504929:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.504938:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.504950:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.504955:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768606.504991:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077014528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505018:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505021:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768606.505023:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505042:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505046:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505049:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505058:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768606.505071:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077014656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505081:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505084:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768606.505086:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505090:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00100000:1.0:1775768606.505095:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077014528:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768606.505096:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505104:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505109:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768606.505122:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077014784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505143:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505147:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768606.505148:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505151:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505156:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768606.505159:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768606.505163:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505168:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768606.505179:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077014912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505187:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505190:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768606.505191:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505194:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505197:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505200:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505204:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768606.505213:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077015040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505222:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505226:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768606.505228:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505232:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505235:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505239:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505254:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768606.505263:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825dc00 x1862023077014528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768606.505267:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077015168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768606.505288:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077014528/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768606.505296:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505304:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768606.505306:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768606.505309:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768606.505313:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505317:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768606.505318:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00080000:3.0:1775768606.505324:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505327:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077014528:192.168.202.158@tcp:250:kworker.0 00000008:00100000:3.0:1775768606.505332:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768606.505337:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014656:192.168.202.158@tcp:38:kworker.0 00000100:00100000:3.0:1775768606.505345:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077015296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768606.505355:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505358:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768606.505360:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505365:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5108 00000800:00000100:1.0:1775768606.505366:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:3.0:1775768606.505373:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505377:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768606.505382:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768606.505384:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825d500 x1862023077014656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505397:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077014656/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505406:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768606.505406:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077015424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768606.505413:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505421:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014656:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768606.505425:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768606.505429:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768606.505431:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768606.505435:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5108 00000100:00080000:3.0:1775768606.505439:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768606.505443:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505447:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014784:192.168.202.158@tcp:38:kworker.0 00000008:00100000:3.0:1775768606.505448:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768606.505463:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077015552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768606.505474:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505487:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825e300 x1862023077014784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505500:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077014784/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505507:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505511:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505514:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768606.505519:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014912:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768606.505537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505547:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825e680 x1862023077014912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505559:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077014912/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505573:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077014912:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768606.505578:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768606.505596:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505605:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825ea00 x1862023077015040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077015040/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505629:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505633:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505635:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768606.505641:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015168:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768606.505658:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825f100 x1862023077015168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505678:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077015168/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505689:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015168:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768606.505697:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768606.505714:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505721:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825c000 x1862023077015296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505728:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077015296/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505733:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505734:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505737:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768606.505740:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015424:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768606.505749:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505754:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825ed80 x1862023077015424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505760:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077015424/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505764:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505766:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505767:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015424:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768606.505778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768606.505787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768606.505791:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768606/real 1775768606] req@ffffa0435825ce00 x1862023077015552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768606.505797:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077015552/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768606.505801:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768606.505803:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768606.505804:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077015552:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768606.568395:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768606.695197:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768606.695608:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768607.527219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768607.591460:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768607.719180:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768607.720929:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768608.551764:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768608.619198:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768608.743228:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768608.743232:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768609.575606:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768609.639766:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768609.767157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768609.767674:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768610.599989:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768610.663654:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768610.791232:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768610.791508:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00080000:3.0:1775768611.623241:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768611.623258:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768611.623264:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768611.623281:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:3.0:1775768611.623298:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768611.623309:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768611.623401:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023077015680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768611.623434:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768611.623438:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768611.623440:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768611.623445:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:3.0:1775768611.623449:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768611.623453:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768611.623466:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768611.623484:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077015808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768611.623495:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768611.623498:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768611.623500:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768611.623504:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:3.0:1775768611.623508:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768611.623513:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768611.623517:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768611.623528:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077015936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768611.623537:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768611.623540:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768611.623541:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768611.623545:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:3.0:1775768611.623549:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768611.623553:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768611.623558:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768611.623569:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077016064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768611.623578:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768611.623582:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768611.623584:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768611.623588:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:3.0:1775768611.623591:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768611.623594:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768611.623599:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768611.623994:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077015680:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775768611.624116:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768611.624172:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa0435825d880 x1862023077015680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768611.624194:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023077015680/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768611.624205:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768611.624210:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768611.624214:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077015680:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768611.624223:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015808:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768611.624247:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768611.624261:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa0435825ce00 x1862023077015808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768611.624274:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077015808/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768611.624283:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768611.624287:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768611.624290:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015808:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768611.624297:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768611.624317:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768611.624328:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa0435825ed80 x1862023077015936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768611.624344:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077015936/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768611.624351:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768611.624354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768611.624358:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077015936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768611.624364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016064:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768611.624382:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768611.624392:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa0435825c000 x1862023077016064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768611.624402:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077016064/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768611.624410:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768611.624414:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768611.624416:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016064:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768611.627348:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768611.627474:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077016192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768611.627520:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768611.627525:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768611.627529:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768611.627535:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:0.0:1775768611.627549:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768611.627558:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768611.627575:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768611.627600:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023077016320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768611.627630:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768611.627634:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768611.627636:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768611.627642:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:0.0:1775768611.627645:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768611.627650:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768611.627655:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768611.627668:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023077016448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768611.627677:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768611.627681:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768611.627682:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768611.627686:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:0.0:1775768611.627690:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768611.627694:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768611.627700:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768611.627711:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023077016576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768611.627720:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768611.627723:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768611.627724:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768611.627728:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5113 00000100:00080000:0.0:1775768611.627732:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768611.627736:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768611.627740:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768611.627750:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023077016704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768611.629986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768611.630121:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768611.630161:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa0435825f100 x1862023077016192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768611.630183:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077016192/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768611.630204:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768611.630210:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768611.630214:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768611.630224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016320:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768611.630250:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768611.630262:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa04349640380 x1862023077016320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768611.630274:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023077016320/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768611.630284:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768611.630289:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768611.630292:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016320:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768611.630300:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768611.630321:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768611.630331:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa04349642680 x1862023077016448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768611.630342:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023077016448/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768611.630350:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768611.630354:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768611.630358:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768611.630364:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016576:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768611.630399:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768611.630411:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa04349640e00 x1862023077016576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768611.630421:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023077016576/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768611.630428:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768611.630431:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768611.630434:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016576:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768611.630439:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768611.630457:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768611.630467:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768611/real 1775768611] req@ffffa04349642300 x1862023077016704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768611.630476:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023077016704/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768611.630484:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768611.630487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768611.630490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077016704:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768611.691070:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768611.815829:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768611.816111:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768612.651953:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768612.711402:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768612.839183:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768612.839978:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768613.671290:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768613.735271:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768613.863149:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768613.863957:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768614.695249:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768614.759293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768614.887381:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768614.887441:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768615.719143:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768615.785626:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768615.911162:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768615.911914:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00080000:0.0:1775768616.745062:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745070:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768616.745074:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745084:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745093:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745100:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768616.745133:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642a00 x1862023077016832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745153:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745157:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768616.745158:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745163:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745166:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745169:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745177:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768616.745188:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642300 x1862023077016960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745197:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745200:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768616.745202:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745205:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745208:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745211:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745214:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768616.745222:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640e00 x1862023077017088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745229:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745232:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768616.745234:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745237:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745240:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745243:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745247:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768616.745254:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349642680 x1862023077017216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745262:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745265:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768616.745266:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745269:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745272:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745275:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745279:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768616.745286:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640380 x1862023077017344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745294:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745307:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768616.745309:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745312:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745315:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745319:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745323:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768616.745332:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641500 x1862023077017472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745341:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745344:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768616.745345:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745349:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745352:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745356:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745360:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768616.745369:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643100 x1862023077017600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745382:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745385:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768616.745386:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745389:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745393:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745396:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745400:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768616.745409:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640a80 x1862023077017728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768616.745417:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768616.745420:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768616.745421:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768616.745425:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5118 00000100:00080000:0.0:1775768616.745428:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768616.745441:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768616.745446:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768616.745456:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641180 x1862023077017856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768616.748446:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077016832:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775768616.748515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748571:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349642a00 x1862023077016832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748591:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642a00 x1862023077016832/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748601:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748605:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748608:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077016832:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768616.748615:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016960:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768616.748637:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748648:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349642300 x1862023077016960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748659:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642300 x1862023077016960/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748668:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748672:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748675:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077016960:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768616.748682:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768616.748701:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748710:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349640e00 x1862023077017088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748733:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640e00 x1862023077017088/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748741:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748744:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748747:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768616.748753:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017216:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768616.748770:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748778:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349642680 x1862023077017216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748789:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349642680 x1862023077017216/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748796:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748799:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748801:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349642680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017216:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768616.748807:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768616.748820:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748828:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349640380 x1862023077017344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748843:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640380 x1862023077017344/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748851:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748854:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748857:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077017344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768616.748863:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017472:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768616.748876:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748885:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349641500 x1862023077017472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748895:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641500 x1862023077017472/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748903:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748906:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748909:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017472:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768616.748914:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768616.748933:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.748942:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349643100 x1862023077017600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.748953:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643100 x1862023077017600/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.748966:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.748969:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.748971:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768616.748977:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017728:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768616.748992:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.749001:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349640a80 x1862023077017728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.749011:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640a80 x1862023077017728/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.749017:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.749020:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.749023:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017728:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768616.749475:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768616.749499:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768616.749510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768616/real 1775768616] req@ffffa04349641180 x1862023077017856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768616.749521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641180 x1862023077017856/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768616.749529:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768616.749532:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768616.749535:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077017856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768616.749594:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768616.807190:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768616.935312:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768616.936148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768617.767220:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768617.840543:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768617.959174:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768617.959699:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768618.791239:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768618.855191:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768618.983335:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768618.983339:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768619.815236:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768619.879223:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768620.007142:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768620.007233:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768620.839172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768620.903351:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768621.031150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768621.031319:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768621.863232:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768621.863955:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.863962:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768621.863966:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.863974:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.863984:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.863992:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768621.864066:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077017984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864098:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864102:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768621.864103:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864108:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864111:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864115:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864125:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768621.864138:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077018112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864160:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864163:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768621.864165:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864168:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864171:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864175:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864456:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768621.864472:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077018240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864488:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864492:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768621.864494:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864498:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864506:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864511:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768621.864522:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077018368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864530:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864533:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768621.864535:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864538:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864541:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864544:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864548:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768621.864557:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077018496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864565:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864568:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768621.864569:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864573:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864576:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864580:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864584:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768621.864593:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077018624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864602:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864604:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768621.864606:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864609:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864613:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864616:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864621:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768621.864629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077018752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768621.864637:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.864640:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768621.864642:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.864645:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.864648:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.864652:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.864656:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768621.864665:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077018880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768621.865798:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077017984:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775768621.865880:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.865920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350930e00 x1862023077017984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.865939:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077017984/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.865948:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.865953:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.865957:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077017984:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768621.865965:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018112:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768621.865990:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.866002:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350932300 x1862023077018112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.866020:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077018112/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.866447:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.866453:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.866456:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018112:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768621.866463:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018240:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768621.866489:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.866501:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350931880 x1862023077018240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.866513:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077018240/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.866521:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.866524:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.866527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768621.866533:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018368:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768621.866555:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.866566:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350932a00 x1862023077018368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.866576:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077018368/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.866589:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.866592:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.866595:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018368:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768621.866601:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768621.866621:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.866631:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350930a80 x1862023077018496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.866643:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077018496/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.866651:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.866654:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.866656:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077018496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768621.866662:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018624:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768621.866998:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768621.867002:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768621.867004:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768621.867009:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5123 00000100:00080000:1.0:1775768621.867014:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768621.867019:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768621.867025:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768621.869960:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077019008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1775768621.870545:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.870574:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350931c00 x1862023077018624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.870590:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077018624/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.870601:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.870606:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.870610:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018624:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768621.870620:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768621.870644:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.870654:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350932680 x1862023077018752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.870666:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077018752/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.874623:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.874632:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.874637:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768621.875698:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018880:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768621.876124:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.876706:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350930000 x1862023077018880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.877616:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077018880/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.877630:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.877636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.877655:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077018880:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768621.878482:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768621.878543:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768621.878564:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768621/real 1775768621] req@ffffa04350930700 x1862023077019008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768621.878578:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077019008/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768621.878587:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768621.878590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768621.878602:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019008:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768621.927765:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768622.055321:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768622.055406:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768622.888072:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768622.951176:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768623.079199:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768623.079291:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768623.913296:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768623.976216:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768624.104104:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768624.104295:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768624.935704:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768625.000730:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768625.127761:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768625.128057:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768625.959203:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768626.023300:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768626.189746:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768626.189832:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768626.983316:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768626.988155:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.988165:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768626.988170:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.988179:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.988190:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.988197:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768626.988249:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077019136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768626.988371:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077019136:192.168.202.158@tcp:250:kworker.0 00000800:00000100:2.0:1775768626.988428:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768626.988457:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.988461:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768626.988463:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.988469:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.988473:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.988477:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.988486:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768626.988498:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077019264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768626.988723:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350930700 x1862023077019136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.988741:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077019136/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.988755:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.988759:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.988762:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077019136:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768626.988785:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019264:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768626.988810:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.988823:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350930000 x1862023077019264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.988835:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077019264/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.988843:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.988858:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.988861:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019264:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768626.988994:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.988998:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768626.989001:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989005:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989010:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989014:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989019:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768626.989060:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077019392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989070:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989074:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768626.989076:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989079:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989083:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989087:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989092:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768626.989102:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077019520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989110:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989113:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768626.989115:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768626.989197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019392:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768626.989296:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.989307:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350932680 x1862023077019392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.989319:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077019392/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.989397:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.989401:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.989405:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768626.989411:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019520:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768626.989430:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.989439:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350931c00 x1862023077019520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.989450:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077019520/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.989457:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.989460:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.989462:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019520:192.168.202.158@tcp:8:kworker.0 00000100:00080000:1.0:1775768626.989496:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989502:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989506:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989512:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768626.989524:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077019648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989543:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989546:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768626.989548:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989552:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989555:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989559:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989564:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768626.989574:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077019776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989582:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989585:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768626.989586:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989589:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989593:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989596:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989600:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768626.989609:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077019904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989628:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989631:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768626.989633:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989636:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5128 00000100:00080000:1.0:1775768626.989640:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989643:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989647:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768626.989767:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019648:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768626.989791:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.989802:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350930a80 x1862023077019648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.989812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077019648/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.989820:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.989823:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.989826:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077019648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768626.989831:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768626.989927:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077020032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768626.989936:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768626.989939:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768626.989941:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768626.989944:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5128 00000800:00000100:2.0:1775768626.989964:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768626.989982:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768626.989987:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768626.989992:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768626.990003:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077020160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768626.990093:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350932a00 x1862023077019776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.990106:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077019776/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.990114:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.990117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.990190:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019776:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768626.990197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019904:192.168.202.158@tcp:38:kworker.0 00000800:00000100:2.0:1775768626.990291:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.990301:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350931880 x1862023077019904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.990313:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077019904/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.990320:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.990323:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.990404:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077019904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768626.990422:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020032:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768626.990442:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.990453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350932300 x1862023077020032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.990464:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077020032/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.990472:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.990475:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.990478:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020032:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768626.990483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020160:192.168.202.158@tcp:8:kworker.0 00000800:00000100:2.0:1775768626.990500:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:2.0:1775768626.990510:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768626/real 1775768626] req@ffffa04350930e00 x1862023077020160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768626.990521:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077020160/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768626.990529:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768626.990532:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768626.990535:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020160:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768627.047196:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768627.239151:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768627.239680:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768628.007194:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768628.076478:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768628.263261:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768628.264770:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768629.032536:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768629.095394:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768629.287170:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768629.287290:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768630.056258:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768630.120630:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768630.311152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768630.311261:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768631.079157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768631.255837:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768631.335462:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768631.342022:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768632.103294:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768632.103459:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.103466:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768632.103469:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.103479:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.103489:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.103495:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768632.103827:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077020288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.103859:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.103863:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768632.103865:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.103869:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.103874:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.103878:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.103887:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768632.103900:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077020416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.103909:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.103912:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768632.103913:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.103916:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.103920:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.103923:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.103926:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768632.103935:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077020544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.103942:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.103944:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768632.103946:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.103948:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.103951:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.103954:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.103958:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768632.103966:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077020672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.103974:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.103977:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768632.103978:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.103982:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.103985:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.103988:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.103991:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768632.103998:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077020800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.104005:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.104008:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768632.104009:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.104012:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.104014:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.104018:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.104021:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768632.104076:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077020928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.104085:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.104088:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768632.104091:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768632.104092:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077020288:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768632.104097:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.104103:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.104107:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.104112:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768632.104123:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077021056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.104133:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.104136:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768632.104137:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.104141:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.104145:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768632.104149:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.104153:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768632.104163:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077021184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768632.104171:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768632.104175:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768632.104177:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768632.104181:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5133 00000100:00080000:3.0:1775768632.104184:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775768632.104187:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020416:192.168.202.158@tcp:38:kworker.0 00000100:00080000:3.0:1775768632.104191:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768632.104198:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768632.104210:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077021312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768632.104224:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768632.104250:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768632.104272:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768632.104296:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768632.104317:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768632.104335:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768632.104352:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021312:192.168.202.158@tcp:8:kworker.0 00010000:00010000:1.0:1775768632.295317:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768632.363747:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768632.371358:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768633.131747:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768633.319177:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768633.383137:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768633.383137:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768634.151673:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768634.343341:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768634.407112:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768634.407154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768635.175157:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:2.0:1775768635.175694:0:2396:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:2.0:1775768635.175737:0:2396:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768635.175753:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175803:0:2396:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175815:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175823:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175831:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175837:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175845:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175851:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:2.0:1775768635.175857:0:2396:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:0.0:1775768635.176254:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825f100 x1862023077020928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176279:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077020928/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176290:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176299:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077020928:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768635.176306:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825c000 x1862023077021056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176318:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077021056/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176324:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176326:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176329:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768635.176336:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825ed80 x1862023077021184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176351:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077021184/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176357:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176359:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176362:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021184:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768635.176367:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825ce00 x1862023077021312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176376:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077021312/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176383:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176386:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176389:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077021312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768635.176395:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825ea00 x1862023077020288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176406:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077020288/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176413:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176415:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176418:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077020288:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768635.176424:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825e680 x1862023077020416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176434:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077020416/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176441:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176444:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176446:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020416:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768635.176451:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825e300 x1862023077020544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176459:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077020544/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176466:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176469:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176471:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768635.176477:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825d500 x1862023077020672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176487:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077020672/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176493:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176496:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176499:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020672:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768635.176504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768632/real 1775768635] req@ffffa0435825dc00 x1862023077020800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768635.176515:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077020800/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768635.176522:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768635.176525:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768635.176527:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077020800:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768635.367701:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768635.431519:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768635.436482:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768636.199138:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768636.391977:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768636.455308:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768636.455593:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:2.0:1775768637.224219:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:0.0:1775768637.226858:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.226870:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768637.226875:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.226909:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.226925:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.226933:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:0.0:1775768637.226992:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077021440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227050:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227055:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768637.227057:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227061:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.227066:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.227070:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.227082:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768637.227098:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077021568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227114:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227118:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768637.227119:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227123:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.227127:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.227130:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.227135:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768637.227145:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077021696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227153:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227157:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768637.227158:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227161:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.227165:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.227169:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.227174:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768637.227184:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077021824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227194:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227197:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768637.227198:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227202:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.227206:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.227210:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.227215:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768637.227365:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077021440:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775768637.227447:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.227493:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349640700 x1862023077021440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.227525:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077021440/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.227536:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.227541:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.227544:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077021440:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768637.227551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021568:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768637.227575:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.227588:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349643b80 x1862023077021568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.227599:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077021568/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.227609:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.227612:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.227615:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021568:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768637.227621:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768637.227639:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.227649:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349643800 x1862023077021696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.227659:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077021696/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.227667:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.227669:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.227672:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768637.227677:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021824:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768637.227696:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.227705:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349641880 x1862023077021824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.227714:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077021824/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.227721:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.227724:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.227726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021824:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768637.227902:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641c00 x1862023077021952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227927:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227931:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768637.227932:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227937:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.227943:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.227947:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.227954:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768637.227970:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349641880 x1862023077022080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.227989:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.227992:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768637.227994:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.227997:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.228001:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.228004:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.228008:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768637.228017:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643800 x1862023077022208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.228055:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.228058:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768637.228060:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.228064:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.228068:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.228070:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.228075:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768637.228085:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349643b80 x1862023077022336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768637.228092:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768637.228095:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768637.228096:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768637.228099:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5138 00000100:00080000:0.0:1775768637.228102:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768637.228105:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:0.0:1775768637.228109:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768637.228117:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04349640700 x1862023077022464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768637.234444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768637.234583:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.234639:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349641c00 x1862023077021952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.234667:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641c00 x1862023077021952/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.234682:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.234688:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.234692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077021952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768637.234700:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022080:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768637.234726:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.234737:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349641880 x1862023077022080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.234748:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349641880 x1862023077022080/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.234758:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.234762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.234765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349641880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022080:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768637.234772:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768637.234792:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.234801:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349643800 x1862023077022208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.234812:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643800 x1862023077022208/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.234822:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.234826:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.234829:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768637.234835:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022336:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768637.234854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.234864:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349643b80 x1862023077022336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.234874:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349643b80 x1862023077022336/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.234881:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.234885:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.234888:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349643b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022336:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768637.234894:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768637.234911:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768637.234920:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768637/real 1775768637] req@ffffa04349640700 x1862023077022464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768637.234931:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04349640700 x1862023077022464/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768637.234938:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768637.234941:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768637.234945:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04349640700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077022464:192.168.202.158@tcp:8:kworker.0 00010000:00010000:3.0:1775768637.415293:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768637.479757:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768637.480736:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768638.251925:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768638.439483:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768638.503165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768638.503165:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768639.271307:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768639.463245:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768639.527179:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768639.528801:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768640.295696:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:1.0:1775768640.492884:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768640.551188:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768640.552759:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768641.319239:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00010000:00010000:3.0:1775768641.511645:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768641.575209:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:2.0:1775768641.575338:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768642.343359:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768642.343446:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.343454:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768642.343458:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.343480:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.343493:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.343502:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768642.343554:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077022592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768642.343626:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077022592:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768642.343705:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:1.0:1775768642.343740:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.343745:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768642.343747:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.343752:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.343757:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.343762:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.343772:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768642.343787:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077022720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768642.343855:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350930e00 x1862023077022592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.343876:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077022592/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.343889:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.343894:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.343898:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077022592:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768642.343928:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022720:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768642.343953:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.343967:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350932300 x1862023077022720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.343982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077022720/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.343992:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.343995:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.343999:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022720:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768642.344072:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344076:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768642.344079:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344083:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344088:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344093:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344099:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768642.344112:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077022848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344122:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344126:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768642.344127:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344131:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344135:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344139:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344144:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768642.344156:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077022976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344165:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344169:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768642.344171:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344174:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344178:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344182:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344187:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768642.344196:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077023104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344211:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344214:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768642.344216:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344220:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344224:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344228:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344232:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768642.344241:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077023232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344250:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344253:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768642.344255:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344259:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344262:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344266:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344270:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768642.344280:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077023360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344289:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344292:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768642.344293:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344297:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344301:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344304:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344309:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768642.344319:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077023488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768642.344327:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768642.344330:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768642.344332:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768642.344335:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5144 00000100:00080000:1.0:1775768642.344339:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768642.344342:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768642.344346:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768642.344356:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077023616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768642.344427:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022848:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768642.344452:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344477:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350931880 x1862023077022848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344496:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077022848/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344505:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344508:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344511:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022848:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768642.344517:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022976:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768642.344537:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344548:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350932a00 x1862023077022976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344559:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077022976/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344567:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344570:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344573:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077022976:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768642.344579:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077023104:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768642.344598:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344608:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350930a80 x1862023077023104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344619:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077023104/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344632:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344636:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344639:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077023104:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768642.344644:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023232:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768642.344663:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344673:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350931c00 x1862023077023232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344684:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077023232/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344692:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344695:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344698:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023232:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768642.344704:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023360:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768642.344722:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344731:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350932680 x1862023077023360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344742:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077023360/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344750:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344753:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344757:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023360:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768642.344762:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023488:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768642.344780:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344790:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350930000 x1862023077023488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344800:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077023488/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344807:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344810:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344813:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023488:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768642.344819:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023616:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768642.344836:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768642.344846:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768642/real 1775768642] req@ffffa04350930700 x1862023077023616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768642.344856:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077023616/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768642.344863:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768642.344866:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768642.344869:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077023616:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768642.599678:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768642.660522:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768642.660595:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768643.367289:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768643.623347:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768643.687149:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768643.688536:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768644.391494:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768644.647261:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768644.711271:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768644.711418:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768645.415415:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768645.673049:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768645.735150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768645.735438:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768646.439201:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768646.705819:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768646.763572:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768646.763734:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768647.463804:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768647.465563:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768647.465574:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465583:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465595:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465603:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768647.465635:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077023744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465677:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465681:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768647.465683:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465686:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465690:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465694:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465702:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768647.465714:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077023872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465723:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465735:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768647.465737:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465741:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465745:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465748:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465752:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768647.465762:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077024000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465771:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465774:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768647.465776:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465779:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465782:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465785:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465790:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768647.465798:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077024128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465807:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465809:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768647.465811:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465814:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465817:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465820:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465824:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768647.465833:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077024256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465842:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465845:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768647.465847:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465850:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465864:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465869:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465874:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768647.465884:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077024384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465898:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465901:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768647.465903:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465906:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465910:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465913:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465917:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768647.465927:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077024512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465935:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465938:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768647.465940:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465943:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465946:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465949:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465954:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768647.465962:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077024640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768647.465970:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768647.465972:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768647.465974:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768647.465977:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5149 00000100:00080000:1.0:1775768647.465981:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768647.465984:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768647.465988:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768647.465998:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077024768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768647.467310:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077023744:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768647.467369:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467405:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350930700 x1862023077023744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467437:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077023744/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467455:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467459:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467463:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077023744:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768647.467470:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077023872:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768647.467492:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467504:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350930000 x1862023077023872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467516:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077023872/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467525:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467528:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467531:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077023872:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768647.467537:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024000:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768647.467554:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467562:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350932680 x1862023077024000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467573:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077024000/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467581:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467584:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467587:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024000:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768647.467593:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024128:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768647.467608:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467616:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350931c00 x1862023077024128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467627:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077024128/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467634:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467637:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467640:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024128:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768647.467646:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024256:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768647.467659:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467668:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350930a80 x1862023077024256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467679:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077024256/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467686:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467689:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467692:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077024256:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768647.467696:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024384:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768647.467712:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467722:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350932a00 x1862023077024384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467744:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077024384/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467752:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467755:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467897:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024384:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768647.467904:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024512:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768647.467926:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.467937:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350931880 x1862023077024512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.467949:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077024512/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.467957:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.467960:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.467963:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024512:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768647.467969:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024640:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768647.468261:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.468272:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350932300 x1862023077024640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.468284:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077024640/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.468291:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.468295:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.468298:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024640:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768647.468304:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024768:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768647.468322:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768647.468331:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768647/real 1775768647] req@ffffa04350930e00 x1862023077024768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768647.468341:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077024768/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768647.468348:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768647.468350:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768647.468353:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077024768:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768647.719219:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768647.783206:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768647.783745:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768648.487182:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768648.743369:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768648.807150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768648.807788:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768649.511474:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768649.767668:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768649.831515:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768649.831627:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768650.538234:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768650.791959:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768650.855242:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768650.855576:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768651.560658:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768651.816724:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768651.879212:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:3.0:1775768651.879410:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768652.583461:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768652.583892:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.583899:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768652.583904:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.583917:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.583929:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.583937:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768652.583988:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023077024896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768652.584082:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077024896:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768652.584161:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768652.584429:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.584434:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768652.584436:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.584441:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.584445:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.584450:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.584462:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768652.584478:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023077025024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.584865:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.584870:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768652.584872:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.584876:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.584881:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.584885:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.584892:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768652.584906:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023077025152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.584917:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.584920:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768652.584921:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.584925:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.584928:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.584933:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.584938:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768652.584948:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023077025280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.584958:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.584961:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768652.584963:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.584966:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.584970:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.584973:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.584977:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768652.584987:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023077025408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.584996:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.584999:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768652.585001:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.585005:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.585008:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.585011:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.585016:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768652.585024:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023077025536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768652.585227:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7e300 x1862023077024896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.585391:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023077024896/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.585403:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.585498:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.585504:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077024896:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768652.585539:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025024:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768652.585584:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.585604:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7c380 x1862023077025024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768652.585794:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.585798:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768652.585800:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.585804:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.585809:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.585813:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.585821:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768652.585836:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023077025664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.585845:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.585848:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768652.585849:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.585853:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.585857:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.585860:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.585865:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768652.585876:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023077025792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768652.585885:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768652.585888:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768652.585890:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768652.585894:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5154 00000100:00080000:2.0:1775768652.585897:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768652.585900:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768652.585905:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768652.585915:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023077025920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768652.585972:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023077025024/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.585982:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.585987:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.585991:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025024:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768652.585998:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025152:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768652.586023:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586216:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7f480 x1862023077025152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586300:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023077025152/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586310:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586314:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586318:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025152:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768652.586324:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025280:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768652.586349:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586371:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7f100 x1862023077025280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586385:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023077025280/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586392:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586396:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586399:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025280:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768652.586405:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025408:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768652.586525:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586536:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7dc00 x1862023077025408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586546:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023077025408/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586554:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586557:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586559:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077025408:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768652.586585:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025536:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768652.586604:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586614:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7d500 x1862023077025536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586711:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023077025536/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586719:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586723:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586726:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025536:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768652.586731:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025664:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768652.586751:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586762:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7c000 x1862023077025664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586773:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023077025664/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586781:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586784:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586787:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025664:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768652.586792:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025792:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768652.586812:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586822:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7e680 x1862023077025792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586834:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023077025792/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586845:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586848:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025792:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768652.586854:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025920:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768652.586872:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768652.586882:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768652/real 1775768652] req@ffffa0434aa7e300 x1862023077025920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768652.586892:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023077025920/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768652.586899:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768652.586902:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768652.586905:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077025920:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768652.839203:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768652.903318:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768652.903363:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768653.611686:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768653.866673:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:3.0:1775768653.927279:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768653.927872:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768654.632498:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768654.887179:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768654.951297:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768654.951418:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768655.655362:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768655.911236:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768655.975973:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768655.976142:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768656.679500:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768656.935731:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768656.999337:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768656.999394:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768657.704632:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.704641:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768657.704646:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.704656:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.704741:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.704750:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768657.704862:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077026048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705141:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705146:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768657.705148:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705153:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705158:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705162:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705171:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768657.705187:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077026176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705198:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705201:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768657.705202:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705206:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705210:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705214:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705218:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768657.705227:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077026304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705236:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705239:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768657.705240:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705244:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705247:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705250:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705255:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768657.705264:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077026432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705272:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705275:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768657.705277:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705280:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705284:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705287:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705291:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768657.705301:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077026560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705323:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705326:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768657.705328:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705332:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705335:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705339:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705343:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768657.705354:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077026688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705362:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705365:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768657.705367:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705370:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705374:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705377:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705381:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768657.705390:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077026816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705398:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705401:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768657.705402:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.705405:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.705408:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.705412:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.705415:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768657.705424:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077026944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768657.705432:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768657.705435:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768657.705436:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768657.706144:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5159 00000100:00080000:1.0:1775768657.706150:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768657.706154:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768657.706160:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768657.706174:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077027072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775768657.706246:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:0.0:1775768657.707787:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077026048:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768657.707853:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.707894:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350930e00 x1862023077026048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.707912:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077026048/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.707923:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.707927:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.707930:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077026048:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768657.707937:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026176:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768657.707958:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.707970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350932300 x1862023077026176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.707979:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077026176/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.707987:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.707991:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.707995:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026176:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768657.708001:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026304:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768657.708020:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708056:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350931880 x1862023077026304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.708067:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077026304/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.708074:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.708077:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.708079:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026304:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768657.708084:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026432:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768657.708104:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708114:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350932a00 x1862023077026432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.708124:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077026432/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.708131:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.708134:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.708138:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026432:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768657.708143:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026560:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768657.708161:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708170:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350930a80 x1862023077026560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.708181:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077026560/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.708190:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.708192:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.708195:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077026560:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768657.708200:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026688:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768657.708218:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708226:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350931c00 x1862023077026688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.708236:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077026688/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.708244:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.708247:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.708249:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026688:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768657.708256:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026816:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768657.708273:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708282:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350932680 x1862023077026816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.708297:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077026816/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.708304:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.708316:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.708320:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026816:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768657.708325:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026944:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768657.708343:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.708352:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350930000 x1862023077026944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.709549:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077026944/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.709560:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.709564:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.709567:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077026944:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768657.709576:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027072:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768657.709611:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768657.709626:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768657/real 1775768657] req@ffffa04350930700 x1862023077027072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768657.710582:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077027072/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768657.710598:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768657.710603:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768657.710606:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027072:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768657.960543:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768658.024150:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768658.024257:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768658.728451:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768658.983629:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768659.047286:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768659.048081:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768659.751142:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768660.007148:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768660.071322:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768660.071635:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768660.775175:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768661.031233:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768661.095246:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768661.095358:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768661.799682:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768662.055166:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768662.119232:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768662.120182:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768662.823787:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:3.0:1775768662.824753:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.824760:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768662.824764:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.824773:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.824783:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.824790:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768662.824826:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023077027200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.824857:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.824861:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768662.824863:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.824866:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.824870:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.824874:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.824883:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768662.824896:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077027328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.824906:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.824909:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768662.824911:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.824914:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.824917:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.824921:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.824925:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768662.824935:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077027456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.824943:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.824947:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768662.824948:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.824951:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.824954:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.824958:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.824962:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768662.824970:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077027584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.824979:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.824982:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768662.824984:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.824987:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.824990:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.824993:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.824997:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768662.825005:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077027712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.825013:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.825016:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768662.825017:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.825020:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.825023:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.825055:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.825061:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768662.825069:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077027840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.825077:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.825079:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768662.825081:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.825084:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.825087:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.825090:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.825094:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768662.825101:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077027968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.825108:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.825110:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768662.825112:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.825115:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.825117:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.825120:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.825124:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768662.825130:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077028096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768662.825137:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768662.825139:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768662.825141:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768662.825143:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5164 00000100:00080000:3.0:1775768662.825145:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768662.825147:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768662.825150:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768662.825156:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077028224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768662.825235:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077027200:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768662.825299:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825360:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825d880 x1862023077027200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825375:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023077027200/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825383:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825387:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825390:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077027200:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768662.825397:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027328:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768662.825417:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825429:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825dc00 x1862023077027328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825440:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077027328/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825448:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825451:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825453:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027328:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768662.825458:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027456:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768662.825476:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825486:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825d500 x1862023077027456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825497:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077027456/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825504:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825507:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825509:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027456:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768662.825514:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027584:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768662.825534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825543:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825e300 x1862023077027584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825553:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077027584/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825559:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825562:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825566:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027584:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768662.825571:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027712:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768662.825586:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825594:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825e680 x1862023077027712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825603:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077027712/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825610:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825613:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825616:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077027712:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768662.825620:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027840:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768662.825634:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825642:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825ea00 x1862023077027840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825651:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077027840/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825659:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825661:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825664:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027840:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768662.825669:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027968:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768662.825682:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825690:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825ce00 x1862023077027968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825699:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077027968/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825706:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825708:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825711:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077027968:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768662.825716:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028096:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768662.825733:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825742:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825ed80 x1862023077028096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825752:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077028096/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825759:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825762:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825765:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028096:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768662.825770:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028224:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768662.825787:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768662.825796:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768662/real 1775768662] req@ffffa0435825c000 x1862023077028224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768662.825807:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077028224/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768662.825815:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768662.825818:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768662.825820:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028224:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768663.080687:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768663.143817:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768663.144173:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768663.848221:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768664.103159:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768664.167191:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768664.172289:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768664.871480:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768665.128570:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768665.195019:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768665.195655:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768665.895156:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768666.154543:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768666.216062:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768666.216326:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768666.919617:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768667.175163:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768667.239617:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768667.239636:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768667.943224:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943237:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768667.943243:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943256:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943271:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943279:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768667.943335:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077028352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943369:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943372:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768667.943375:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943379:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943383:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943387:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943398:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768667.943413:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077028480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943423:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943426:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768667.943427:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943432:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943435:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943439:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943444:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768667.943453:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077028608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943462:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943465:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768667.943467:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943470:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943473:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943477:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943482:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768667.943493:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077028736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943500:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943503:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768667.943505:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943508:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943511:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943514:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943518:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768667.943528:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077028864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943536:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943539:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768667.943541:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943545:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943548:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943552:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943556:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768667.943565:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077028992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943573:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943576:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768667.943577:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943581:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943584:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943588:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943592:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768667.943600:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077029120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943608:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943610:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768667.943612:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943615:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943618:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943622:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943627:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768667.943636:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077029248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768667.943645:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768667.943647:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768667.943649:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768667.943666:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5169 00000100:00080000:3.0:1775768667.943669:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768667.943672:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768667.943677:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768667.943687:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077029376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768667.944062:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077028352:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768667.944141:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944192:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825f100 x1862023077028352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944224:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077028352/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944237:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944241:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944245:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077028352:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768667.944253:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028480:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768667.944277:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944290:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825c000 x1862023077028480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077028480/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944310:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944312:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944316:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028480:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768667.944321:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028608:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768667.944338:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944347:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825ed80 x1862023077028608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944358:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077028608/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944365:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944368:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944371:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028608:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768667.944376:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028736:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768667.944392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944400:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825ce00 x1862023077028736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944410:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077028736/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944419:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944422:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944425:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028736:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768667.944430:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028864:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768667.944445:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944453:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825ea00 x1862023077028864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944463:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077028864/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944471:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944474:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944477:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077028864:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768667.944483:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028992:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768667.944501:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944512:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825e680 x1862023077028992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944523:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077028992/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944531:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944534:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944537:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077028992:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768667.944542:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029120:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768667.944560:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944570:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825e300 x1862023077029120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944580:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077029120/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944587:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944590:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944592:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029120:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768667.944597:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029248:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768667.944614:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944623:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825d500 x1862023077029248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944634:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077029248/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944641:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944645:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944648:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029248:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768667.944667:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029376:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768667.944686:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768667.944696:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768667/real 1775768667] req@ffffa0435825dc00 x1862023077029376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768667.944707:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077029376/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768667.944714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768667.944717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768667.944720:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077029376:192.168.202.158@tcp:8:kworker.0 00000100:00100000:3.0:1775768667.946080:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768668.199121:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768668.263213:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768668.264642:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768668.967162:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768669.226052:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768669.287471:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768669.288129:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768669.991609:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768670.247293:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768670.311689:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:0.0:1775768670.311975:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768671.015320:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768671.271447:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768671.335156:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768671.335480:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768672.047769:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768672.295265:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:0.0:1775768672.359636:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768672.359713:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768673.064126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768673.064552:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064559:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768673.064563:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064573:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064584:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064590:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768673.064629:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077029504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064665:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064674:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768673.064676:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064680:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064684:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064687:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064695:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768673.064708:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077029632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064716:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064719:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768673.064721:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064725:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064728:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064731:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064736:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768673.064746:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077029760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064755:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064758:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768673.064760:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064773:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064777:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064782:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064787:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768673.064797:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077029888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064805:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064808:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768673.064809:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064812:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064816:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064819:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064823:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768673.064832:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077030016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064839:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064842:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768673.064843:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064847:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064850:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064854:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064858:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768673.064878:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077030144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064887:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064890:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768673.064891:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064902:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064905:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064908:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064913:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768673.064922:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077030272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064930:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064933:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768673.064935:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064938:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064941:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064944:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064948:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768673.064957:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077030400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768673.064964:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768673.064967:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768673.064969:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768673.064972:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5174 00000100:00080000:1.0:1775768673.064974:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768673.064977:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768673.064981:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768673.065000:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077030528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768673.065886:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077029504:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768673.065947:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.065996:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350930700 x1862023077029504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066013:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077029504/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066023:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066047:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066052:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077029504:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768673.066059:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029632:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768673.066082:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066095:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350930000 x1862023077029632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066107:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077029632/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066114:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066117:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066120:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029632:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768673.066125:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029760:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768673.066159:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066171:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350932680 x1862023077029760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066180:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077029760/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066187:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066190:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066192:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029760:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768673.066197:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029888:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768673.066215:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066224:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350931c00 x1862023077029888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066234:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077029888/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066253:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066256:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066258:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077029888:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768673.066264:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030016:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768673.066282:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066291:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350930a80 x1862023077030016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066301:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077030016/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066308:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066311:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066314:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030016:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768673.066319:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030144:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768673.066336:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066346:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350932a00 x1862023077030144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066357:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077030144/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066364:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066367:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066370:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030144:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768673.066375:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030272:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768673.066392:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350931880 x1862023077030272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077030272/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066418:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066421:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066424:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030272:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768673.066429:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030400:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768673.066448:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066458:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350932300 x1862023077030400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066468:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077030400/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066476:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066479:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066482:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030400:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768673.066488:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030528:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768673.066505:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768673.066515:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768673/real 1775768673] req@ffffa04350930e00 x1862023077030528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768673.066524:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077030528/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768673.066532:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768673.066545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768673.066549:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077030528:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768673.319241:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768673.383209:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768673.383354:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768674.087902:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768674.343216:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768674.407851:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768674.409215:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768675.113126:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768675.367646:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768675.431271:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768675.431631:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768676.138277:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768676.392153:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768676.455192:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768676.455651:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768677.159628:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768677.418545:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768677.479263:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768677.479351:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768678.183168:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768678.186265:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186276:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768678.186280:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186291:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186302:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186309:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768678.186348:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7fb80 x1862023077030656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186381:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186385:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768678.186387:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186391:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186395:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186399:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186409:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768678.186422:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c700 x1862023077030784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186431:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186434:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768678.186435:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186439:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186442:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186446:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186450:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768678.186459:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ce00 x1862023077030912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186468:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186471:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768678.186472:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186475:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186478:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186481:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186486:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768678.186495:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7df80 x1862023077031040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186503:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186506:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768678.186508:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186511:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186514:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186517:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186521:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1775768678.186530:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ed80 x1862023077031168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186538:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186541:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768678.186543:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186546:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186549:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186553:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186557:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768678.186566:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d880 x1862023077031296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186574:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186577:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768678.186579:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186581:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186584:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186587:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186592:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768678.186600:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7ea00 x1862023077031424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186608:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186610:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768678.186612:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186615:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186617:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186620:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186624:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768678.186632:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f800 x1862023077031552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768678.186638:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768678.186641:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768678.186642:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768678.186645:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5179 00000100:00080000:2.0:1775768678.186648:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768678.186650:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768678.186654:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768678.186663:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023077031680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768678.186754:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077030656:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768678.186827:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.186878:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7fb80 x1862023077030656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.186894:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7fb80 x1862023077030656/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.186909:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.186913:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.186916:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077030656:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768678.186924:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030784:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768678.186945:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.186956:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7c700 x1862023077030784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.186966:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c700 x1862023077030784/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.186974:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.186977:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.186981:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030784:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768678.186986:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030912:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768678.187003:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187012:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7ce00 x1862023077030912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187022:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ce00 x1862023077030912/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187053:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187056:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187059:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077030912:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768678.187065:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031040:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768678.187084:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187094:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7df80 x1862023077031040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187117:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7df80 x1862023077031040/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187125:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187128:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187130:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031040:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768678.187135:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031168:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768678.187156:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187165:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7ed80 x1862023077031168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187175:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ed80 x1862023077031168/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187183:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187188:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031168:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768678.187193:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031296:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768678.187210:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187219:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7d880 x1862023077031296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187228:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d880 x1862023077031296/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187235:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187238:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187240:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031296:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768678.187245:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031424:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768678.187259:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187267:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7ea00 x1862023077031424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187277:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7ea00 x1862023077031424/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187283:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187286:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187289:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031424:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768678.187294:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031552:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768678.187310:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187319:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7f800 x1862023077031552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187328:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f800 x1862023077031552/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187340:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187343:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187346:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031552:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768678.187351:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031680:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768678.187367:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768678.187376:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768678/real 1775768678] req@ffffa0434aa7e300 x1862023077031680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768678.187387:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023077031680/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768678.187394:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768678.187396:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768678.187399:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077031680:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768678.439262:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768678.503228:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768678.514217:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768679.207239:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768679.463165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:2.0:1775768679.528139:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768679.531836:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:0.0:1775768680.231378:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768680.487170:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768680.551349:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768680.551941:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768681.255150:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768681.512260:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768681.575251:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768681.575378:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768682.280899:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768682.535201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768682.599169:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768682.602667:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768683.306269:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306278:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768683.306283:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306293:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306304:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306311:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768683.306345:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d880 x1862023077031808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306370:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306374:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768683.306376:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306379:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306383:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306387:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306395:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768683.306407:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077031936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306416:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306419:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768683.306420:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306424:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306426:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306429:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306433:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768683.306440:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077032064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306447:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306451:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768683.306452:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306455:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306457:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306459:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306463:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768683.306469:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077032192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306476:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306479:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768683.306481:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306484:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306487:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306490:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306495:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768683.306505:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077032320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306513:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306516:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768683.306518:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306522:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306525:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306529:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306534:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768683.306542:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077032448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306550:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306552:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768683.306553:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306557:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306559:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306562:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306566:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768683.306573:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077032576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306580:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306583:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768683.306584:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306587:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306591:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306593:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306597:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768683.306604:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077032704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768683.306611:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768683.306613:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768683.306615:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768683.306618:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5184 00000100:00080000:3.0:1775768683.306621:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768683.306624:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768683.306628:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768683.306636:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077032832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768683.307276:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:1.0:1775768683.309062:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077031808:192.168.202.158@tcp:250:kworker.0 00000800:00000100:1.0:1775768683.309147:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309188:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825d880 x1862023077031808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309207:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d880 x1862023077031808/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309218:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309222:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309226:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077031808:192.168.202.158@tcp:250:kworker.0 00000100:00100000:1.0:1775768683.309233:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031936:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768683.309271:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309283:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825dc00 x1862023077031936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309295:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077031936/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309303:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309307:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309309:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077031936:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768683.309315:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032064:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768683.309334:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309343:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825d500 x1862023077032064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309354:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077032064/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309361:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309364:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309367:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032064:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768683.309373:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032192:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768683.309391:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309401:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825e300 x1862023077032192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077032192/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309424:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309427:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309430:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032192:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768683.309436:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032320:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768683.309454:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309464:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825e680 x1862023077032320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309475:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077032320/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309484:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309487:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309490:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077032320:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768683.309496:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032448:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768683.309514:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309523:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825ea00 x1862023077032448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077032448/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309542:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309545:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309548:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032448:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768683.309554:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032576:192.168.202.158@tcp:38:kworker.0 00000800:00000100:1.0:1775768683.309572:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309581:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825ce00 x1862023077032576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309592:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077032576/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309599:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309602:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309605:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032576:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768683.309611:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032704:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768683.309628:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309638:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825ed80 x1862023077032704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309649:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077032704/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309660:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032704:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768683.309667:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032832:192.168.202.158@tcp:8:kworker.0 00000800:00000100:1.0:1775768683.309685:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:1.0:1775768683.309695:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768683/real 1775768683] req@ffffa0435825c000 x1862023077032832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768683.309706:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077032832/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768683.309714:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768683.309717:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768683.309719:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077032832:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768683.559289:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768683.623359:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768683.623496:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768684.327177:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768684.583553:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768684.647246:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768684.647867:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768685.351715:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768685.607156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768685.671281:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768685.671351:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768686.375144:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768686.632450:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768686.695127:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768686.696354:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768687.399210:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768687.655247:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768687.719158:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768687.719336:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768688.423337:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768688.423467:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423473:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768688.423476:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423486:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423496:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423503:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768688.423543:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077032960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423576:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423580:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768688.423582:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423586:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423589:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423593:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423602:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768688.423612:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077033088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423622:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423624:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768688.423626:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423628:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423631:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423633:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423637:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768688.423643:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077033216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423648:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423650:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768688.423651:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423653:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423655:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423656:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423659:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768688.423663:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077033344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423668:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423670:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768688.423671:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423674:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423676:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423678:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423681:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768688.423685:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077033472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423690:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423693:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768688.423694:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423697:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423699:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423703:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423707:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768688.423716:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077033600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423723:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.423726:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768688.423727:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.423731:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.423733:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.423736:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.423741:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768688.423749:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077033728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.423757:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768688.423966:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077032960:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768688.424042:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424081:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350930e00 x1862023077032960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424097:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077032960/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424110:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424113:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077032960:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768688.424121:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033088:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768688.424140:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424148:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350932300 x1862023077033088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424155:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077033088/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424161:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424162:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424165:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033088:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768688.424168:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033216:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768688.424178:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424183:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350931880 x1862023077033216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424189:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077033216/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424193:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424194:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424196:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033216:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768688.424199:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033344:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768688.424208:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424213:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350932a00 x1862023077033344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424219:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077033344/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424224:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424235:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424238:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033344:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768688.424241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033472:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768688.424251:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424257:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350930a80 x1862023077033472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424267:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077033472/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424274:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424277:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424279:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077033472:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768688.424285:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033600:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768688.424302:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424311:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350931c00 x1862023077033600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424322:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077033600/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424327:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424329:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424331:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033600:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768688.424335:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033728:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768688.424350:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424358:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350932680 x1862023077033728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424369:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077033728/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424376:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424379:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424382:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033728:192.168.202.158@tcp:38:kworker.0 00000100:00080000:1.0:1775768688.424525:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768688.424527:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.424531:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.424535:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.424538:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.424543:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768688.424553:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077033856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768688.424569:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768688.424572:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768688.424573:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768688.424576:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5190 00000100:00080000:1.0:1775768688.424578:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768688.424580:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768688.424582:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768688.424588:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077033984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768688.424636:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033856:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768688.424653:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424661:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350930000 x1862023077033856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424668:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077033856/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424673:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424675:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424677:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033856:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768688.424680:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033984:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768688.424689:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768688.424697:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768688/real 1775768688] req@ffffa04350930700 x1862023077033984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768688.424708:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077033984/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768688.424715:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768688.424718:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768688.424721:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077033984:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768688.679165:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768688.744535:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768688.745654:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768689.447386:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768689.705919:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768689.767727:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768689.768097:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768690.472110:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768690.727210:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768690.791591:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768690.792510:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768691.495278:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768691.751198:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768691.815177:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768691.816220:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768692.519232:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768692.775888:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768692.839146:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768692.839542:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768693.543243:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:2.0:1775768693.543325:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543331:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768693.543335:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543345:0:52808:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543355:0:52808:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543363:0:52808:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:2.0:1775768693.543401:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023077034112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543420:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543426:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768693.543429:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543434:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543438:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768693.543444:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077034112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768693.543445:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543457:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768693.543469:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c000 x1862023077034240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543480:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543483:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768693.543485:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543489:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543493:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543497:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543501:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768693.543512:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7d500 x1862023077034368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543520:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543523:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768693.543525:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543529:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5195 00000800:00000100:0.0:1775768693.543534:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768693.543535:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543542:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543547:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1775768693.543558:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7dc00 x1862023077034496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543568:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543571:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768693.543573:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543577:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543581:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543585:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543590:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1775768693.543601:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7e680 x1862023077034112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768693.543603:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f100 x1862023077034624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543630:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543633:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768693.543635:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768693.543638:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023077034112/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768693.543642:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543648:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543655:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00080000:0.0:1775768693.543657:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768693.543661:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775768693.543673:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:2.0:1775768693.543674:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7f480 x1862023077034752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768693.543681:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077034112:192.168.202.158@tcp:250:kworker.0 00000100:00080000:2.0:1775768693.543688:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543692:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768693.543694:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543699:0:52808:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543703:0:52808:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543707:0:52808:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543712:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768693.543715:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:2.0:1775768693.543726:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7c380 x1862023077034880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543736:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543739:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768693.543741:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543745:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543749:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768693.543755:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00080000:2.0:1775768693.543756:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543763:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1775768693.543774:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7c000 x1862023077034240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768693.543778:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e680 x1862023077035008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768693.543804:0:52808:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768693.543807:0:52808:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768693.543810:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c000 x1862023077034240/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768693.543810:0:52808:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768693.543818:0:52808:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5195 00000100:00080000:2.0:1775768693.543823:0:52808:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768693.543826:0:52808:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:2.0:1775768693.543830:0:52808:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1775768693.543842:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775768693.543842:0:52808:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0434aa7e300 x1862023077035136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768693.543852:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.543859:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034240:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768693.543867:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034368:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768693.543890:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.543900:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7d500 x1862023077034368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.543911:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7d500 x1862023077034368/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.543919:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.543922:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.543925:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034368:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768693.543931:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034496:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768693.543959:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.543970:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7dc00 x1862023077034496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.543981:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7dc00 x1862023077034496/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.543989:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.543992:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.543995:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034496:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768693.544001:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034624:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768693.544020:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.544051:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7f100 x1862023077034624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.544063:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f100 x1862023077034624/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.544070:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.544073:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.544076:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077034624:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768693.544081:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077034752:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768693.544098:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.544109:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7f480 x1862023077034752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.544120:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7f480 x1862023077034752/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.544127:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.544130:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.544133:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077034752:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768693.544154:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077034880:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768693.544171:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.544181:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7c380 x1862023077034880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.544192:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7c380 x1862023077034880/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.544199:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.544203:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.544205:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077034880:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768693.544210:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035008:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768693.544227:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.544238:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7e680 x1862023077035008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.544249:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e680 x1862023077035008/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.544256:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.544259:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.544261:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035008:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768693.544267:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035136:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768693.544283:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768693.544292:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768693/real 1775768693] req@ffffa0434aa7e300 x1862023077035136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768693.544302:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0434aa7e300 x1862023077035136/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768693.544309:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768693.544312:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768693.544314:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0434aa7e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035136:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768693.799131:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768693.863278:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768693.863429:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768694.569320:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768694.823195:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768694.887145:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768694.887339:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768695.591101:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768695.847201:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768695.911168:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768695.911546:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768696.615151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768696.871150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768696.935170:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768696.935281:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768697.640865:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768697.895241:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768697.959189:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768697.959834:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768698.663278:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663287:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768698.663292:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663300:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663310:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663316:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:3.0:1775768698.663753:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825f100 x1862023077035264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.663784:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663788:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768698.663790:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663794:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663798:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663802:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.663812:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768698.663826:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825c000 x1862023077035392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768698.663834:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077035264:192.168.202.158@tcp:250:kworker.0 00000100:00080000:3.0:1775768698.663838:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663841:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768698.663842:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663846:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663848:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663851:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.663854:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768698.663862:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ed80 x1862023077035520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.663869:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663871:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768698.663873:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663876:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663878:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663881:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.663885:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768698.663891:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ce00 x1862023077035648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.663909:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663911:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768698.663913:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663915:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663918:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663920:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.663924:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1775768698.663945:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825ea00 x1862023077035776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.663953:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663957:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768698.663959:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663962:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.663966:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.663970:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.663974:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768698.663984:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e680 x1862023077035904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.663991:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.663994:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768698.663995:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.663998:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.664001:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.664003:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.664007:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768698.664013:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825e300 x1862023077036032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.664020:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.664023:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768698.664024:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.664046:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.664049:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.664052:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.664056:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768698.664064:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825d500 x1862023077036160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768698.664073:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768698.664076:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768698.664077:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768698.664082:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5200 00000100:00080000:3.0:1775768698.664084:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768698.664087:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:3.0:1775768698.664091:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1775768698.664099:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa0435825dc00 x1862023077036288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768698.664139:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768698.664165:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768698.664181:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768698.664208:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768698.664226:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768698.664241:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768698.664257:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768698.664273:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768698.664549:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768698.920292:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768698.983143:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768698.985281:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768699.688506:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768699.943382:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768700.007615:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768700.007857:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768700.711151:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768700.967160:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768701.031355:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768701.031904:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768701.736663:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000400:00000100:0.0:1775768701.744293:0:2398:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.158:988 00000400:00000100:0.0:1775768701.744373:0:2398:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.158@tcp at host 192.168.202.158 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768701.744394:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744457:0:2398:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744470:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744486:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744495:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744502:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744509:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744516:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000800:00000100:0.0:1775768701.744523:0:2398:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.158@tcp 00000100:00100000:1.0:1775768701.746325:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825c000 x1862023077035392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746357:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825c000 x1862023077035392/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746370:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746375:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746391:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035392:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768701.746400:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825ed80 x1862023077035520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746411:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ed80 x1862023077035520/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746419:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746421:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746426:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035520:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768701.746432:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825ce00 x1862023077035648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746442:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ce00 x1862023077035648/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746448:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746451:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746453:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035648:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768701.746459:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825ea00 x1862023077035776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746469:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825ea00 x1862023077035776/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746486:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746493:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077035776:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768701.746500:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825e680 x1862023077035904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746510:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e680 x1862023077035904/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746517:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746521:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746523:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077035904:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768701.746546:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825e300 x1862023077036032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746558:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825e300 x1862023077036032/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746564:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746567:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746570:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036032:192.168.202.158@tcp:38:kworker.0 00000100:00100000:1.0:1775768701.746576:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825d500 x1862023077036160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746587:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825d500 x1862023077036160/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746594:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746597:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746600:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036160:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768701.746607:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825dc00 x1862023077036288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746618:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825dc00 x1862023077036288/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746624:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746627:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746630:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077036288:192.168.202.158@tcp:8:kworker.0 00000100:00100000:1.0:1775768701.746637:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768698/real 1775768701] req@ffffa0435825f100 x1862023077035264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768701.746648:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa0435825f100 x1862023077035264/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768701.746655:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768701.746658:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:1.0:1775768701.746662:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa0435825f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077035264:192.168.202.158@tcp:250:kworker.0 00000100:00100000:2.0:1775768701.991150:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768702.055115:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768702.057502:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768702.761193:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768703.015156:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768703.080284:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768703.099832:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768703.783165:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768703.783793:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.783810:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768703.783815:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.783825:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.783837:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.783866:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768703.783904:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077036416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.783941:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.783946:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768703.783947:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.783951:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.783955:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.783959:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.783969:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768703.783982:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077036544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.783992:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.783997:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768703.783998:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784002:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784005:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784009:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784014:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768703.784024:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077036672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784080:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784084:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768703.784085:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784089:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784092:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784095:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784101:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768703.784112:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077036800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784120:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784123:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768703.784124:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784129:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784132:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784136:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784502:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768703.784518:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077036928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784527:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784530:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768703.784532:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784535:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784540:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784543:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784549:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768703.784558:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077037056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784567:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784570:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768703.784572:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784575:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784578:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784582:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784586:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768703.784595:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077037184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784603:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784606:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768703.784608:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784611:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784615:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784618:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784623:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768703.784631:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077037312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768703.784639:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768703.784642:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768703.784644:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768703.784647:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5205 00000100:00080000:1.0:1775768703.784651:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768703.784654:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768703.784658:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768703.784667:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077037440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768703.784778:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077036416:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768703.784868:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.784912:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350930700 x1862023077036416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.784929:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077036416/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.784939:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.784943:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.784946:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077036416:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768703.784954:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036544:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768703.784975:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.784987:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350930000 x1862023077036544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.784998:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077036544/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785006:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785010:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785013:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036544:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768703.785019:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036672:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768703.785064:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785086:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350932680 x1862023077036672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785098:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077036672/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785106:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785108:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785111:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036672:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768703.785116:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036800:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768703.785136:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785297:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350931c00 x1862023077036800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785312:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077036800/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785318:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785321:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785323:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036800:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768703.785329:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036928:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768703.785350:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785359:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350930a80 x1862023077036928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785368:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077036928/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785381:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785384:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785387:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077036928:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768703.785391:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037056:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768703.785408:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785417:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350932a00 x1862023077037056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785427:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077037056/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785435:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785438:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785441:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037056:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768703.785446:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037184:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768703.785462:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785470:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350931880 x1862023077037184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785480:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077037184/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785488:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785490:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785494:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037184:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768703.785499:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037312:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768703.785515:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785524:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350932300 x1862023077037312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785534:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077037312/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785541:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785543:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785546:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037312:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768703.785551:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037440:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768703.785567:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768703.785577:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768703/real 1775768703] req@ffffa04350930e00 x1862023077037440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768703.785587:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077037440/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768703.785593:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768703.785596:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768703.785598:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077037440:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768704.042729:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768704.103404:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768704.103965:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768704.807976:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768705.063154:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768705.127157:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768705.127418:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768705.831107:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768706.087940:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768706.151930:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768706.153510:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768706.855376:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768707.111152:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00010000:00010000:1.0:1775768707.175477:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768707.176091:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00000100:00100000:3.0:1775768707.879136:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768708.135231:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:3.0:1775768708.200453:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:2.0:1775768708.200617:0:34677:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768708.903303:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00080000:1.0:1775768708.903372:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3bc78d45-5379-4d62-8e28-79875d287077->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903377:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768708.903380:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903387:0:52508:0:(import.c:509:import_select_connection()) MGC192.168.202.158@tcp: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903402:0:52508:0:(import.c:588:import_select_connection()) MGC192.168.202.158@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903409:0:52508:0:(import.c:623:import_select_connection()) MGC192.168.202.158@tcp: import ffffa043509e9800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000100:00100000:1.0:1775768708.903440:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930e00 x1862023077037568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903462:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903464:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768708.903465:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903468:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903470:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903473:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352a18800: import ffffa0435095a000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903480:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768708.903488:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932300 x1862023077037696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903494:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903496:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768708.903497:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903500:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903502:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903504:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352a18800: import ffffa04350958000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903506:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768708.903511:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931880 x1862023077037824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903515:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903517:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768708.903518:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903520:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903522:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903524:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352a18800: import ffffa0435095b800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903526:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768708.903531:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932a00 x1862023077037952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903535:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2bb64567-481f-43c2-999a-8c9c071338ae->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903537:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768708.903538:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903540:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903541:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903543:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352a18800: import ffffa0435095e800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903545:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1775768708.903549:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930a80 x1862023077038080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903561:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903565:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768708.903566:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903570:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903573:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903577:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffffa04352624800: import ffffa04350ded800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903580:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768708.903589:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350931c00 x1862023077038208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903597:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903600:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768708.903601:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903604:0:52508:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903607:0:52508:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903610:0:52508:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffffa04352624800: import ffffa04350dee000 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903614:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768708.903621:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350932680 x1862023077038336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903629:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903631:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768708.903632:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903636:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903639:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903642:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffffa04352624800: import ffffa04350dea800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903646:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768708.903654:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930000 x1862023077038464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768708.903661:0:52508:0:(pinger.c:229:ptlrpc_pinger_process_import()) 11b6e21a-ce9d-4bf4-8cc8-5d154b257af0->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768708.903665:0:52508:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768708.903666:0:52508:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768708.903669:0:52508:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: connect to NID 192.168.202.158@tcp last attempt 5210 00000100:00080000:1.0:1775768708.903671:0:52508:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768708.903674:0:52508:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffffa04352624800: import ffffa04350dee800 using connection 192.168.202.158@tcp/192.168.202.158@tcp 00000008:00100000:1.0:1775768708.903678:0:52508:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1775768708.903687:0:52508:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffffa04350930700 x1862023077038592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768708.903756:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077037568:192.168.202.158@tcp:250:kworker.0 00000800:00000100:0.0:1775768708.903854:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.903902:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350930e00 x1862023077037568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.903917:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930e00 x1862023077037568/t0(0) o250->MGC192.168.202.158@tcp@192.168.202.158@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.903927:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa043509e9800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.903930:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.903934:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3bc78d45-5379-4d62-8e28-79875d287077:2407:1862023077037568:192.168.202.158@tcp:250:kworker.0 00000100:00100000:0.0:1775768708.903941:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037696:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768708.903961:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.903972:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350932300 x1862023077037696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.903982:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932300 x1862023077037696/t0(0) o38->lustre-MDT0000-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.903990:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.903993:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.903996:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037696:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768708.904001:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037824:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768708.904017:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904043:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350931880 x1862023077037824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904056:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931880 x1862023077037824/t0(0) o38->lustre-MDT0001-mdc-ffffa04352a18800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904064:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350958000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904067:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904070:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037824:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768708.904075:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037952:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768708.904091:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904100:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350932a00 x1862023077037952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904110:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932a00 x1862023077037952/t0(0) o8->lustre-OST0000-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904118:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095b800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904121:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904124:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077037952:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768708.904129:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077038080:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768708.904143:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904151:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350930a80 x1862023077038080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904173:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930a80 x1862023077038080/t0(0) o8->lustre-OST0001-osc-ffffa04352a18800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904182:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa0435095e800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904185:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904188:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2bb64567-481f-43c2-999a-8c9c071338ae:2407:1862023077038080:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768708.904193:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038208:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768708.904210:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904219:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350931c00 x1862023077038208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904230:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350931c00 x1862023077038208/t0(0) o38->lustre-MDT0000-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904238:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350ded800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904241:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904244:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350931c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038208:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768708.904249:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038336:192.168.202.158@tcp:38:kworker.0 00000800:00000100:0.0:1775768708.904263:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904271:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350932680 x1862023077038336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904282:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350932680 x1862023077038336/t0(0) o38->lustre-MDT0001-mdc-ffffa04352624800@192.168.202.158@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904289:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904291:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904295:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350932680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038336:192.168.202.158@tcp:38:kworker.0 00000100:00100000:0.0:1775768708.904300:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038464:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768708.904319:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904328:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350930000 x1862023077038464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904338:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930000 x1862023077038464/t0(0) o8->lustre-OST0000-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904345:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dea800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904347:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904350:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038464:192.168.202.158@tcp:8:kworker.0 00000100:00100000:0.0:1775768708.904355:0:2407:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038592:192.168.202.158@tcp:8:kworker.0 00000800:00000100:0.0:1775768708.904371:0:2407:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.158@tcp 00000100:00100000:0.0:1775768708.904380:0:2407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768708/real 1775768708] req@ffffa04350930700 x1862023077038592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768708.904390:0:2407:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffffa04350930700 x1862023077038592/t0(0) o8->lustre-OST0001-osc-ffffa04352624800@192.168.202.158@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768708.904396:0:2407:0:(import.c:65:import_set_state_nolock()) ffffa04350dee800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768708.904399:0:2407:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.158@tcp failed (-110) 00000100:00100000:0.0:1775768708.904402:0:2407:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffffa04350930700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:11b6e21a-ce9d-4bf4-8cc8-5d154b257af0:2407:1862023077038592:192.168.202.158@tcp:8:kworker.0 00000100:00100000:2.0:1775768709.159177:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds 00000100:00100000:0.0:1775768709.225055:0:51207:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043594cf480 going to sleep for 0 seconds 00010000:00010000:1.0:1775768709.225148:0:6179:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775768709.929172:0:6655:0:(client.c:2691:ptlrpc_set_wait()) set ffffa043541989c0 going to sleep for 0 seconds 00000100:00100000:2.0:1775768710.183290:0:52330:0:(client.c:2691:ptlrpc_set_wait()) set ffffa04354198a80 going to sleep for 0 seconds Debug log: 79208 lines, 79208 kept, 0 dropped, 0 bad.